AJPv13 extensions Proposal

Apache Software Foundation | Jakarta Project | Apache Tomcat

## AJPv13 extensions Proposal

print-friendly
version

Introduction

This document is a proposal of evolution of the current Apache JServ Protocol version 1.3, also known as ajp13. I'll not cover here the full protocol but only the add-on from ajp13. This nth pass include comments from the tomcat-dev list and misses discovered during developpment.

Missing features in AJP13

ajp13 is a good protocol to link a servlet engine like tomcat to a web server like Apache:

- use persistants connections to avoid reconnect time at each request
- encode many http commands to reduce stream size
- send to servlet engine many info from web server (like SSL certs)

- security between web server and servlet engine. Anybody can connect to an ajp13 port (no login mecanism used) You could connect, for example with telnet, and keep the remote thread up by not sending any data (no timeout in connection)
- context information passed from servlet engine to web server. Part of the configuration of JK, the web server connector, is to indicate to the web server which URI to handle. The mod_jk JkMount directive, told to web server which URI must be forwarded to servlet engine. A servlet engine allready knows which URI it handle and TC 3.3 is allready capable to generate a config file for JK from the list of available contexts.
- state update of contexts from servlet engine to web server. Big site with farm of Tomcat, like ISP and virtuals hosters, may need to stop a context for admin purposes. In that case the front web server must know that the context is currently down, to eventually relay the request to another Tomcat
- verify state of connection before sending request. Actually JK send the request to the servlet engine and next wait for the answer. But one of the beauty of the socket API, is you that you could write() to a closed connection without any error reporting, but a read() to a closed connection return you the error code.

Proposed add-ons to AJP13

Let's descrive here the features and add-on that could be added to AJP13. Since this document is a proposal, a reasonable level of chaos must be expected at first. Be sure that discussion on tomcat list will help clarify points, add features but the current list seems to be a 'minimun vital'

- Advanced login features at connect time
- Basic authorisation system, where a shared secret key is present in web server and servlet engine.
- Basic protocol negociation, just to be sure that if functionnalities are added to AJP13 in the future, current implementations will still works.
- Clean handling of 'Unknown packets'
- Extended env vars passed from web-server to servlet engine.
- Add extra SSL informations needed by Servlet 2.3 API (like SSL_KEY_SIZE)

Advanced login

1. WEB-SERVER send LOGIN INIT CMD + NEGOCIATION DATA + WEB SERVER INFO
2. TOMCAT respond with LOGIN SEED CMD + RANDOM DATA
3. WEB-SERVER calculted the MD5 of RANDOM DATA+SECRET DATA
4. WEB-SERVER send LOGIN COMP CMD + MD5 (SECRET DATA + RANDOM DATA)
5. TOMCAT respond with LOGIN STATUS CMD + NEGOCIED DATA + SERVLET ENGINE INFO

To prevent DOS attack, the servlet engine will wait the LOGIN CMD only 15/30 seconds and reports the timeout exception for admins investigation. The login command will contains basic protocol negociation information like compressing ability, crypto, context info (at start up), context update at run-time (up/down), level of SSL env vars, AJP protocol level supported (level1/level2/level3...) The Web server info will contain web server info and connector name (ie Apache 1.3.26 + mod_ssl 2.8.8 + mod_jk 1.2.1 + mod_perl 1.25). The servlet engine will mask the negociation mask with it's own mask (what it can do) and return it when loggin is accepted. This will help having a basic AJP13 implementation (level 1) on a web-server working with a more advanced protocol handler on the servlet engine side or vice-versa. AJP13 was designed to be small and fast and so many SSL informations present in the web-server are not forwarded to the servlet engine. We add here four negociations flags to provide more informations on client SSL data (certs), server SSL datas, crypto used, and misc datas (timeout...).

Messages Stream

    +----------------+------------------+-----------------+
    | LOGIN INIT CMD | NEGOCIATION DATA | WEB SERVER INFO |
    +----------------+------------------+-----------------+

    +----------------+----------------+
    | LOGIN SEED CMD | MD5 of entropy |
    +----------------+----------------+

    +----------------+----------------------------+
    | LOGIN COMP CMD | MD5 of RANDOM + SECRET KEY |
    +----------------+----------------------------+

    +-----------+---------------+---------------------+
    | LOGOK CMD | NEGOCIED DATA | SERVLET ENGINE INFO |
    +-----------+---------------+---------------------+

    +------------+--------------+
    | LOGNOK CMD | FAILURE CODE |
    +------------+--------------+

- LOGIN INIT CMD, LOGIN SEED CMD, LOGIN COMP CMD, LOGOK CMD, LOGNOK CMD are 1 byte long.
- MD5, MD5 of RANDOM + SECRET KEY are 32 chars long.
- NEGOCIATION DATA, NEGOCIED DATA, FAILURE CODE are 32 bits long.
- WEB SERVER INFO, SERVLET ENGINE INFO are CString.

The secret key will be set by a new propertie in workers.properties : secretkey

    worker.ajp13.port=8009
    worker.ajp13.host=localhost
    worker.ajp13.type=ajp13
    worker.ajp13.secretkey=myverysecretkey

Shutdown feature

AJP13 miss a functionnality of AJP12, which is shutdown command. A logout will tell servlet engine to shutdown itself.

    +--------------+----------------------------+
    | SHUTDOWN CMD | MD5 of RANDOM + SECRET KEY |
    +--------------+----------------------------+

    +------------+
    | SHUTOK CMD |
    +------------+

    +-------------+--------------+
    | SHUTNOK CMD | FAILURE CODE |
    +-------------+--------------+

- SHUTDOWN CMD, SHUTOK CMD, SHUTNOK CMD are 1 byte long.
- MD5 of RANDOM + SECRET KEY are 32 chars long.
- FAILURE CODE is 32 bits long.

Extended Env Vars feature

NOTA: While working on AJP13 in JK, I really discovered "JkEnvVar". The following "Extended Env Vars feature" description may not be implemented in extended AJP13 since allready available in original implementation. DESC: Many users will want to see some of their web-server env vars passed to their servlet engine. To reduce the network traffic, the web-servlet will send a table to describing the external vars in a shorter fashion. We'll use there a functionnality allready present in AJP13, attributes list : In the AJP13, we've got :

    AJP13_FORWARD_REQUEST :=
        prefix_code      2
        method           (byte)
        protocol         (string)
        req_uri          (string)
        remote_addr      (string)
        remote_host      (string)
        server_name      (string)
        server_port      (integer)
        is_ssl           (boolean)
        num_headers      (integer)
        request_headers *(req_header_name req_header_value)

        ?context       (byte string)
        ?servlet_path  (byte string)
        ?remote_user   (byte string)
        ?auth_type     (byte string)
        ?query_string  (byte string)
        ?jvm_route     (byte string)
        ?ssl_cert      (byte string)
        ?ssl_cipher    (byte string)
        ?ssl_session   (byte string)

        ?attributes   *(attribute_name attribute_value)
        request_terminator (byte)

Using short 'web server attribute name' will reduce the network traffic.

    +-------------------+---------------------------+-------------------------------+----+
    | EXTENDED VARS CMD | WEB SERVER ATTRIBUTE NAME | SERVLET ENGINE ATTRIBUTE NAME | ES |
    +-------------------+---------------------------+-------------------------------+----+

ie :

    JkExtVars S1 SSL_CLIENT_V_START javax.servlet.request.ssl_start_cert_date
    JkExtVars S2 SSL_CLIENT_V_END   javax.servlet.request.ssl_end_cert_date
    JkExtVars S3 SSL_SESSION_ID     javax.servlet.request.ssl_session_id

    +-------------------+----+-------------------------------------------+
    | EXTENDED VARS CMD | S1 | javax.servlet.request.ssl_start_cert_date |
    +-------------------+----+-------------------------------------------+
    +----+-----------------------------------------+
    | S2 | javax.servlet.request.ssl_end_cert_date |
    +----+-----------------------------------------+
    +----+-----------------------------------------+
    | S3 | javax.servlet.request.ssl_end_cert_date |
    +----+-----------------------------------------+

During transmission in extended AJP13 we'll see attributes name containing S1, S2, S3 and attributes values of 2001/01/03, 2002/01/03, 0123AFE56. This example showed the use of extended SSL vars but any 'personnal' web-server vars like custom authentification vars could be reused in the servlet engine. The cost will be only some more bytes in the AJP traffic.

- EXTENDED VARS CMD is 1 byte long.
- WEB SERVER ATTRIBUTE NAME, SERVLET ENGINE ATTRIBUTE NAME are CString.
- ES is an empty CString.

Context informations forwarding for Servlet engine to Web Server

Just after the LOGON PHASE, the web server will ask for the list of contexts and URLs/URIs handled by the servlet engine. It will ease installation in many sites, reduce questions about configuration on tomcat-user list, and be ready for servlet API 2.3. This mode will be activated by a new directive JkAutoMount ie: JkAutoMount examples myworker1 /examples/ If we want to get ALL the contexts handled by the servlet engine, willcard could be used : ie: JkAutoMount * myworker1 * A servlet engine could have many contexts, /examples, /admin, /test. We may want to use only some contexts for a given worker. It was done previously, in apache HTTP server for example, by setting by hand the JkMount accordingly in each [virtual] area of Apache. If you web-server support virtual hosting, we'll forward also that information to servlet engine which will only return contexts for that virtual host. In that case the servlet engine will only return the URL/URI matching these particular virtual server (defined in server.xml). This feature will help ISP and big sites which mutualize large farm of Tomcat in load-balancing configuration.

    +-----------------+-------------------+----------+----------+----+
    | CONTEXT QRY CMD | VIRTUAL HOST NAME | CONTEXTA | CONTEXTB | ES |
    +-----------------+-------------------+----------+----------+----+

    +------------------+-------------------+----------+-------------------+----------+---------------+----+
    | CONTEXT INFO CMD | VIRTUAL HOST NAME | CONTEXTA | URL1 URL2 URL3 ES | CONTEXTB | URL1 URL2 ... | ES |
    +------------------+-------------------+----------+-------------------+----------+---------------+----+

We'll discover via context-query, the list of URL/MIMES handled by the remove servlet engine for a list of contextes. In wildcard mode, CONTEXTA will contains just '*'.

- CONTEXT QRY CMD and CONTEXT INFO CMD are 1 byte long.
- VIRTUAL HOST NAME is a CString, ie an array of chars terminated by a null byte (/0).
- An empty string is just a null byte (/0).
- ES is an empty CString. Indicate end of URI/URLs or end of CONTEXTs.

NB:
When VirtualMode is not to be used, the VIRTUAL HOST NAME is '*'. In that case the servlet engine will send all contexts handled.

Context informations updates from Servlet engine to Web Server

Context update are messages caming from the servlet engine each time a context is desactivated/reactivated. The update will be in use when the directive JkUpdateMount. This directive will set the AJP13_CONTEXT_UPDATE_NEG flag. ie: JkUpdateMount myworker1

    +--------------------+-------------------+----------+--------+----------+--------+----+
    | CONTEXT UPDATE CMD | VIRTUAL HOST NAME | CONTEXTA | STATUS | CONTEXTB | STATUS | ES |
    +--------------------+-------------------+----------+--------+----------+--------+----+

- CONTEXT UPDATE CMD, STATUS are 1 byte long.
- VIRTUAL HOST NAME, CONTEXTS are CString.
- ES is an empty CString. Indicate end of CONTEXTs.

NB:
When VirtualMode is not in use, the VIRTUAL HOST NAME is '*'. STATUS is one byte indicating if context is UP/DOWN/INVALID

Context status query to Servlet engine

This query will be used by the web-server to determine if a given contexts are UP, DOWN or INVALID (and should be removed).

    +-------------------+--------------------+----------+----------+----+
    | CONTEXT STATE CMD |  VIRTUAL HOST NAME | CONTEXTA | CONTEXTB | ES |
    +-------------------+--------------------+----------+----------+----+

    +-------------------------+-------------------+----------+--------+----------+--------+----+
    | CONTEXT STATE REPLY CMD | VIRTUAL HOST NAME | CONTEXTA | STATUS | CONTEXTB | STATUS | ES |
    +-------------------------+-------------------+----------+-------------------+--------+----+

- CONTEXT STATE CMD, CONTEXT STATE REPLY CMD, STATUS are 1 byte long.
- VIRTUAL HOST NAME, CONTEXTs are CString
- ES is an empty CString

NB:
When VirtualMode is not in use, the VIRTUAL HOST NAME is an empty string.

Handling of unknown packets

Sometimes even with a well negocied protocol, we may be in a situation where one end (web server or servlet engine), will receive a message it couldn't understand. In that case the receiver will send an 'UNKNOW PACKET CMD' with attached the unhandled message.

    +--------------------+------------------------+-------------------+
    | UNKNOWN PACKET CMD | UNHANDLED MESSAGE SIZE | UNHANDLED MESSAGE |
    +--------------------+------------------------+-------------------+

Depending on the message, the sender will report an error and if possible will try to forward the message to another endpoint.

- UNKNOWN PACKET CMD is 1 byte long.
- UNHANDLED MESSAGE SIZE is 16bits long.
- UNHANDLED MESSAGE is an array of byte (length is contained in UNHANDLED MESSAGE SIZE)

NB:
added UNHANDLED MESSAGE SIZE (development)

Verification of connection before sending request

NOTA: This fonctionality may never be used, since it may slow up the normal process since requiring on the web-server side an extra IO (read) before forwarding the request..... One of the beauty of socket APIs, is that you could write on a half closed socket. When servlet engine close the socket, the web server will discover it only at the next read() to the socket. Basically, in the AJP13 protocol, the web server send the HTTP HEADER and HTTP BODY (POST by chunk of 8K) to the servlet engine and then try to receive the reply. If the connection was broken the web server will learn it only at receive time. We could use a buffering scheme but what happen when you use the servlet engine for upload operations with more than 8ko of datas ? The hack in the AJP13 protocol is to add some bytes to read after the end of the service :

    EXAMPLE OF DISCUSSION BETWEEN WEB SERVER AND SERVLET ENGINE

    AJP HTTP-HEADER (+ HTTP-POST)   (WEB->SERVLET)

    AJP HTTP-REPLY					(SERVLET->WEB)

    AJP END OF DISCUSSION			(SERVLET->WEB)

    ---> AJP STATUS 				(SERVLET->WEB AJP13)

The AJP STATUS will not be read by the servlet engine at the end of the request/response #N but at the begining of the next session. More at that time the web server could also use OS dependants functions (or better APR functions) to determine if there is also more data to read. And that datas could be CONTEXT Updates. This will avoid the web server sending a request to a desactivated context. In that case, if the load-balancing is used, it will search for another servlet engine to handle the request. And that feature will help ISP and big sites with farm of tomcat, to updates their servlet engine without any service interruption.

    +------------+-------------+
    | STATUS CMD | STATUS DATA |
    +------------+-------------+

- STATUS CMD and STATUS DATA are one byte long.

Conclusion

The goal of the extended AJP13 protocol is to overcome some of the original AJP13 limitation. An easier configuration, a better support for large site and farm of Tomcat, a simple authentification system and provision for protocol updates. Using the stable ajp13 implementation in JK (native) and in servlet engine (java), it's a reasonable evolution of the well known ajp13.

Commands and IDs in extended AJP13 Index

Index of Commands and ID to be added in AJP13 Protocol

Commands IDs

Command Name | Command Number
AJP13_LOGINIT_CMD | 0x10
AJP13_LOGSEED_CMD | 0x11
AJP13_LOGCOMP_CMD | 0x12
AJP13_LOGOK_CMD | 0x13
AJP13_LOGNOK_CMD | 0x14
AJP13_CONTEXT_QRY_CMD | 0x15
AJP13_CONTEXT_INFO_CMD | 0x16
AJP13_CONTEXT_UPDATE_CMD | 0x17
AJP13_STATUS_CMD | 0x18
AJP13_SHUTDOWN_CMD | 0x19
AJP13_SHUTOK_CMD | 0x1A
AJP13_SHUTNOK_CMD | 0x1B
AJP13_CONTEXT_STATE_CMD | 0x1C
AJP13_CONTEXT_STATE_REP_CMD | 0x1D
AJP13_UNKNOW_PACKET_CMD | 0x1E

Negociations Flags

Command Name | Number | Description
AJP13_CONTEXT_INFO_NEG | 0x80000000 | web-server want context info after login
AJP13_CONTEXT_UPDATE_NEG | 0x40000000 | web-server want context updates
AJP13_GZIP_STREAM_NEG | 0x20000000 | web-server want compressed stream
AJP13_DES56_STREAM_NEG | 0x10000000 | web-server want crypted DES56 stream with secret key
AJP13_SSL_VSERVER_NEG | 0x08000000 | Extended info on server SSL vars
AJP13_SSL_VCLIENT_NEG | 0x04000000 | Extended info on client SSL vars
AJP13_SSL_VCRYPTO_NEG | 0x02000000 | Extended info on crypto SSL vars
AJP13_SSL_VMISC_NEG | 0x01000000 | Extended info on misc SSL vars

Negociation ID | Number | Description
AJP13_PROTO_SUPPORT_AJPXX_NEG | 0x00FF0000 | mask of protocol supported
AJP13_PROTO_SUPPORT_AJP13L1_NEG | 0x00010000 | communication could use AJP13 Level 1
AJP13_PROTO_SUPPORT_AJP13L2_NEG | 0x00020000 | communication could use AJP13 Level 2
AJP13_PROTO_SUPPORT_AJP13L3_NEG | 0x00040000 | communication could use AJP13 Level 3

All others flags must be set to 0 since they are reserved for future use.

Failure IDs

Failure Id | Number
AJP13_BAD_KEY_ERR | 0xFFFFFFFF
AJP13_ENGINE_DOWN_ERR | 0xFFFFFFFE
AJP13_RETRY_LATER_ERR | 0xFFFFFFFD
AJP13_SHUT_AUTHOR_FAILED_ERR | 0xFFFFFFFC

Status

Failure Id | Number
AJP13_CONTEXT_DOWN | 0x01
AJP13_CONTEXT_UP | 0x02
AJP13_CONTEXT_OK | 0x03