Hadoop encryption key deletion – permanently

Hadoop key is used to encrypt and decrypt the data stored in HDFS directories, Here am going to describe how to create, delete, undelete and purge(permanently delete) the hadoop key.

So first let’s create a key. For creating key you need to be a super user of hadoop.

Throughout this post, you are going to see, my key name as encryption-keyEncryption key name.

1. Create Key

hadoop key create encryption-key

Suppose say, you have created the wrong key-name or later you want to change the key name.

Delete the key using hadoop delete command, will only hide the key, this will not permanently delete the key from the client.

After deleting the key and list, you may not be able to see, it just hides the key and you can get back(undelete) the key at any time.

2. Deleting the key

hadoop key delete encryption-key

By issuing the above command, you can delete the key and list it using below command.

3. List available keys

hadoop key list

If you list the key after deleting it, you will not be able to see the deleted by. But that doesn’t mean that you key is deleted. To confirm that, create key with the same name [Step 1], then you will get an exception saying that, “Key exists but has been disabled“.

Exception:

[boopathi@boopathi.me ~]$ hadoop key create encryption-key
encryption-key has not been created. java.io.IOException: HTTP status [500], exception [com.cloudera.keytrustee.TrusteeKeyProvider$DuplicateKeyException], message [Key with name "encryption-key" already exists in "com.cloudera.keytrustee.TrusteeKeyProvider@4c156b4b. Key exists but has been disabled. Use undelete to enable.]
java.io.IOException: HTTP status [500], exception [com.cloudera.keytrustee.TrusteeKeyProvider$DuplicateKeyException], message [Key with name "encryption-key" already exists in "com.cloudera.keytrustee.TrusteeKeyProvider@4c156b4b. Key exists but has been disabled. Use undelete to enable.]
        at org.apache.hadoop.util.HttpExceptionUtils.validateResponse(HttpExceptionUtils.java:159)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:545)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:503)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKeyInternal(KMSClientProvider.java:676)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKey(KMSClientProvider.java:684)
        at org.apache.hadoop.crypto.key.KeyShell$CreateCommand.execute(KeyShell.java:483)
        at org.apache.hadoop.crypto.key.KeyShell.run(KeyShell.java:79)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:515)
[boopathi@boopathi.me ~]$

It is very clear that, the key is just hidden and disabled in the client, but not deleted. If you want to use the same key, then undelete the key. Use the below command to do.

4. Undelete Hadoop key

#Non-Kerberized environment :

curl -L -d "trusteeOp=undelete" "http://<kms host name>:16000/kms/v1/trustee/key/encryption-key?user.name=<username>&trusteeOp=undelete"

#Kerberized environment
curl -L --negotiate -u <username> -d "trusteeOp=undelete" "http://<kms hostname>:16000/kms/v1/trustee/key/encryption-key?user.name=<username>&trusteeOp=undelete"

Once you are done with undeleting the key, now execute the key list command to see the key listed in the client. Now we have come to the expected steps to delete the key permanently or purge it. So here we call it as purge the key. Check out the below command to purge the key in both Kerberized and Non-Kerberized environment.

5. Purge or Permanently delete the key

#Non-Kerberized environment

curl -L -d "trusteeOp=purge" http://<hostname>:16000/kms/v1/trustee/key/encryption-key?user.name=<username>&trusteeOp=purge

#Kerberized environment

curl -L --negotiate -u <username> -d "trusteeOp=purge" http://<hostname>:16000/kms/v1/trustee/key/encryption-key?user.name=<username>&trusteeOp=purge

Now, you key has been deleted permanently or purged successfully, to check that do the list [step 3] and undelete the key again[step 4]. When you are trying to undelete the purged key, you will get an “encryption-key” not found exception.

Exception:

{
 "RemoteException" : {
 "message" : "Key with name encryption-key not found in com.cloudera.keytrustee.TrusteeKeyProvider@7829d789. ",
 "exception" : "IOException",
 "javaClassName" : "java.io.IOException"
   }
}

Now, you have confirmed that, your key has been removed from the client permanently, you can go ahead and create new key.

Here you find document for creating encryption Zone