Information for Netatalk Developers

For basic installation instructions, see the Installation chapter in the html
manual published on https://netatalk.io and the INSTALL.md file in the root of
the source tree.

Netatalk is an implementation of Apple Filing Protocol (AFP) over TCP. The
session layer used to carry AFP over TCP is called DSI.

Netatalk also supports the AppleTalk Protocol Suite for legacy Macs, Lisas and
Apple IIs via the "atalkd" daemon. It supports EtherTalk Phase I and II, RTMP,
NBP, ZIP, AEP, ATP, PAP, and ASP, while expecting the kernel to supply DDP.

The complete stack looks like this on a BSD-derived system:

AFP                          AFP
 |                            |
ASP    PAP                   DSI
  \   /                       |
   ATP RTMP NBP ZIP AEP       |
    |    |   |   |   |        |
-+---------------------------------------------------+- (kernel boundary)
|                    Socket                         |
+-----------------------+------------+--------------+
|                       |     TCP    |    UDP       |
|          DDP          +------------+--------------+
|                       |           IP              |
+-----------------------+---------------------------+
|                Network-Interface                  |
+---------------------------------------------------+

  - DDP is a socket to socket protocol that all other AppleTalk protocols are
    built on top of.

  - "atalkd" implements RTMP, NBP, ZIP, and AEP. It is the AppleTalk equivalent
    of Unix "routed".

  - There is also a client-stub library for NBP. ATP and ASP are implemented as
    libraries.

  - "papd" allows Macs to spool to "lpd", and "pap" allows Unix machines to
    print to AppleTalk connected printers.

When built without AppleTalk support, the network stack looks something like
this:

          AFP
           |
          DSI
           |
           | (port:548)
           |
   -+---------------------------+- (kernel boundary)
    |         Socket            |
    +------------+--------------+
    |     TCP    |    UDP       |
    +------------+--------------+
    |       IP v4 or v6         |
    +---------------------------+
    |     Network Interface     |
    +---------------------------+

Error checking and logging

We want rigid error checking and concise log messages. This often leads to
significant code bloat where the relevant function call is buried in error
checking and logging statements. In order to alleviate error checking and code
readability, we provide a set of error checking macros in <atalk/errchk.h>.
These macros compare the return value of statements against 0, NULL, -1 (and
maybe more, check it out). Every macro comes in four flavours: EC_CHECK,
EC_CHECK_LOG, EC_CHECK_LOG_ERR and EC_CHECK_CUSTOM:

  - EC_CHECK just checks the CHECK
  - EC_CHECK_LOG additionally logs the stringified function call.
  - EC_CHECK_LOG_ERR allows specifying the return value
  - EC_CHECK_CUSTOM allows custom actions

The macros EC_CHECK* unconditionally jump to a cleanup label where the necessary
cleanup can be done alongside controlling the return value. EC_CHECK_CUSTOM
doesn't do that, so an extra "goto EC_CLEANUP" may be performed as appropriate.

Example:

- stat() without EC macro:
  static int func(const char *name) {
    int ret = 0;
    ...
    if ((ret = stat(name, &some_struct_stat)) != 0) {
      LOG(...);
      ret = -1; /* often needed to explicitly set the error indicating return value */
      goto cleanup;
    }

    return ret;

  cleanup:
    ...
    return ret;
  }

- stat() with EC macro:
  static int func(const char *name) {
    EC_INIT; /* expands to int ret = 0; */

    char *uppername = NULL
    EC_NULL(uppername = strdup(name));
    EC_ZERO(strtoupper(uppername));

    EC_ZERO(stat(uppername, &some_struct_stat)); /* expands to complete if block from above */

    EC_STATUS(0);

EC_CLEANUP:
    if (uppername) free(uppername);
    EC_EXIT;
  }

A boilerplate function template is:

int func(void)
{
    EC_INIT;

    ...your code here...

    EC_STATUS(0);

EC_CLEANUP:
    EC_EXIT;
}

CNID Database Daemons

The CNID database daemons cnid_metad and cnid_dbd are an implementation of the
netatalk CNID database support that attempts to put all functionality into
separate daemons. There is one cnid_dbd daemon per netatalk volume. The
underlying database structure is based on Berkeley DB.

Advantages

  - No locking issues or leftover locks due to crashed afpd daemons any more.
    Since there is only one thread of control accessing the database, no locking
    is needed and changes appear atomic.

  - Berkeley DB transactions are difficult to get right with several processes
    attempting to access the CNID database simultaneously. This is much easier
    with a single process and the database can be made nearly crash-proof this
    way (at a performance cost).

  - No problems with user permissions and access to underlying database files,
    the cnid_dbd process runs under a configurable user ID that normally also
    owns the underlying database and can be contacted by whatever afpd daemon
    accesses a volume.

  - If an afpd process crashes, the CNID database is unaffected. If the process
    was making changes to the database at the time of the crash, those changes
    will be rolled back entirely (transactions). If the process was not using
    the database at the time of the crash, no corrective action is necessary. In
    any case, database consistency is assured.

Disadvantages

  - Performance in an environment of processes sharing the database (files) is
    potentially better for two reasons:
    
    i) IPC overhead. ii) r/o access to database pages is possible by more than
    one process at once, r/w access is possible for non overlapping regions.
    
    The current implementation of cnid_dbd uses unix domain sockets as the IPC
    mechanism. While this is not the fastest possible method, it is very
    portable and the cnid_dbd IPC mechanisms can be extended to use faster IPC
    (like mmap) on architectures where it is supported. As a ballpark
    figure, 20000 requests/replies to the cnid_dbd daemon take about 0.6 seconds
    on a Pentium III 733 MHz running Linux Kernel 2.4.18 using unix domain
    sockets. The requests are "empty" (no database lookups/changes), so this is
    just the IPC overhead.
    
    I have not measured the effects of the advantages of simultaneous database
    access.

Installation and configuration

There are two executables that will be built in etc/cnid_dbd and installed into
the systems binaries directories of netatalk cnid_metad and cnid_dbd. cnid_metad
should run all the time with root permissions. It will be notified when an
instance of afpd starts up and will in turn make sure that a cnid_dbd daemon is
started for the volume that afpd wishes to access. The cnid_dbd daemon runs as
long as necessary and services any other instances of afpd that access the
volume. You can safely kill it with SIGTERM, it will be restarted automatically
by cnid_metad as soon as the volume is accessed again.

cnid_dbd changes to the Berkeley DB directory on startup and sets effective UID
and GID to owner and group of that directory. Database and supporting files
should therefore be writeable by that user/group.

Current shortcomings:

  - The parameter file parsing of db_param is very simpleminded. It is easy to
    cause buffer overruns and the like. Also, there is no support for blanks (or
    weird characters) in filenames for the usock_file parameter.

  - There is no protection against a malicious user connecting to the cnid_dbd
    socket and changing the database.

Spotlight Compatible Indexing

Starting with version 3.1 Netatalk supports Spotlight searching. Netatalk uses
GNOME TinySPARQL/LocalSearch (previously known as Tracker) as metadata store,
indexer and search engine.

Limitations and notes

  - Large filesystems
    
    Tracker on Linux uses the inotify Kernel filesystem change event API for
    tracking filesystem changes. On large filesystems this may be problematic
    since the inotify API doesn't offer recursive directory watches but instead
    requires that for every subdirectory watches must be added individually.
    
    On Solaris the FEN file event notification system is used. It is unknown
    which limitations and resource consumption this Solaris subsystem may have.
    
    We therefore recommend to disable live filesystem monitoring and let Tracker
    periodically scan filesystems for changes instead, see the Tracker
    configuration options enable-monitors and crawling-interval below.

  - Indexing home directories
    
    A known limitation with the current implementation means that shared volumes
    in a user's home directory does not get indexed by Spotlight.
    
    As a workaround, keep the shared volumes you want to have indexed elsewhere
    on the host filesystem.

Supported metadata attributes

The following table lists the supported Spotlight metadata attributes, based on
Apple's MDItem object Core Services file metadata standard.

| Description | Spotlight Key |
| --- | --- |
| Name | kMDItemDisplayName, kMDItemFSName |
| Document content (full text search) | kMDItemTextContent |
| File type | _kMDItemGroupId, kMDItemContentTypeTree |
| File modification date | kMDItemFSContentChangeDate,
kMDItemContentModificationDate, kMDItemAttributeChangeDate |
| Content Creation date | kMDItemContentCreationDate |
| The author, or authors, of the contents of the file | kMDItemAuthors,
kMDItemCreator |
| The name of the country where the item was created | kMDItemCountry |
| Duration | kMDItemDurationSeconds |
| Number of pages | kMDItemNumberOfPages |
| Document title | kMDItemTitle |
| The width, in pixels, of the contents. For example, the image width or the
video frame width | kMDItemPixelWidth |
| The height, in pixels, of the contents. For example, the image height or the
video frame height | kMDItemPixelHeight |
| The color space model used by the document contents | kMDItemColorSpace |
| The number of bits per sample | kMDItemBitsPerSample |
| Focal length of the lens, in millimeters | kMDItemFocalLength |
| ISO speed | kMDItemISOSpeed |
| Orientation of the document. Possible values are 0 (landscape) and 1
(portrait) | kMDItemOrientation |
| Resolution width, in DPI | kMDItemResolutionWidthDPI |
| Resolution height, in DPI | kMDItemResolutionHeightDPI |
| Exposure time, in seconds | kMDItemExposureTimeSeconds |
| The composer of the music contained in the audio file | kMDItemComposer |
| The musical genre of the song or composition | kMDItemMusicalGenre |
