Apache Hadoop HDFS

The Apache Hadoop HDFS plugin provides cluster connections, path lookup and policy management in GrantForge. Its plugin ID and service type are both hdfs.

Installation

The distribution includes the plugin in plugins/hdfs. Confirm that hdfs is enabled under Platform management → Plugins; rescan the plugin directory after updating it.

Add a data service

1. Open Data permissions → Data services, add a service and select HDFS (hdfs).
2. Enter the cluster URI and lookup user. Hadoop 2.x uses webhdfs://namenode:50070; 3.x can use hdfs://namenode:8020 or webhdfs://namenode:9870. For HTTPS, use swebhdfs:// with the cluster's actual port.
3. Set the lookup directory and test the connection, which verifies that the directory exists and can be listed, then save the service.

With the NameNode attribute extension enabled in Hadoop 2.7.7, ordinary users querying the root path / trigger a confirmed upstream NullPointerException; set lookup.path to an actual directory such as /data (see the agent guide).

Kerberos also requires a reachable KDC, the server's krb5.conf, and matching hadoop.security.auth_to_local and service principals. The lookup account retrieves directory metadata. Tested against Kerberos clusters of all six lines: lookups over swebhdfs (the service's own truststore and SPNEGO) on every line, and lookups and browsing over RPC (keytab or password) on 3.x; wrong credentials, a missing truststore and a simple client all fail, and the NameNode's own permissions apply to the lookup account.

With Kerberos, GrantForge reuses a sign-in between lookups instead of asking the KDC every time: Hadoop renews a keytab sign-in when its ticket nears its end, and a password sign-in is repeated once less than a fifth of its ticket's life (and at least a minute) remains; a changed password or an updated keytab file signs in again. Each service uses its own truststore, which an ssl-client.xml on the server's class path does not replace.

One GrantForge can reach clusters of several Kerberos realms: give each service the KDCs of its realm in kerberos.kdc; the realm is the one of the principal in username, such as grantforge@BETA.EXAMPLE. GrantForge writes a configuration that includes the server's own krb5.conf and adds the services' realms; it takes effect on the next sign-in, without a restart. With hadoop.security.auth_to_local empty, the realm's principals get short-name rules. The setting cannot be used while the server sets java.security.krb5.realm and .kdc. When two services give different KDCs for the same realm, the most recent lookup wins, so keep them the same; cross-realm trust still belongs in the server's krb5.conf.

Within one GrantForge, connection tests, lookups and browsing of HDFS services run one at a time, never in parallel: Hadoop keeps the authentication type, auth_to_local and the Kerberos sign-in state for the whole process, so concurrent calls would overwrite each other's. Each call is limited by grantforge.plugins.call-timeout (10 seconds by default), time spent waiting its turn included; raise it when there are many services, slow clusters or a slow first Kerberos sign-in.

Configuration is validated on save: in hadoop.config, fs.defaultFS and fs.default.name are aliases, so configure only one; the cluster URI must not contain credentials, a path, a query or a fragment; kerberos requires a password or a keytab; lookup.path must be an absolute path without ..; lookup.max.entries must be between 1 and 100000. Additional properties override connection settings with the same name, and both validation and sign-in use the overridden values.

Browse paths

Select the service under Data permissions → Policies and use Browse beside path.

• Open directories, navigate the path or return to the parent, and load subsequent pages as needed.
• Inspect file/directory markers, owner, group, permissions, file size and modification time.
• Select multiple files or directories, or the current directory, to add to the policy; typing paths still offers suggestions.

RPC and WebHDFS endpoints supporting batched listings use native pagination. Older endpoints read directories within the scan limit and report an error above it. Permission, authentication and connection failures show their reason and can be retried.

Input may omit the leading /, repeated / and . are allowed, and .. and absolute paths outside lookup.path are rejected; a non-existent directory returns no candidates. Browsing does not replace HDFS's own access control; symbolic links and ViewFS mounts still follow the cluster configuration.

Enforce policies

The single resource level is path, with read, write and execute access types. Path policies support recursion and exclusions.

The server plugin handles management and lookups. Enforcing data access also requires a NameNode agent matching the Hadoop version; users must satisfy both native HDFS permissions and GrantForge policies. Numbered agents cover 2.7, 2.10, 3.2, 3.3, 3.4 and 3.5. See the agent guide for verified combinations, authentication and HA coverage, and superuser limitations.

Related guides

• Data services, policies and agents
• Apache Hadoop HDFS NameNode agent
• Plugin and service type development