The Syncrepl Client’s Syncrepl Class

The Syncrepl class is the main class that you will be using. It is responsible for all LDAP operations, and will issue callbacks to an object you provide (see the BaseCallback class).

class syncrepl_client.Syncrepl(data_path, callback, mode, ldap_url=None, starttls=False, **kwargs)

This class implements the Syncrepl client. You should have one instance of this class for each syncrepl search.

Each class requires several items, which will be discussed here:

• A data store

  The Syncrepl client stores a copy of all LDAP records returned by the LDAP server. This data is stored on disk to speed up synchronization if the client loses connection to the LDAP server (either intentionally or not).

  The Syncrepl class writes to several files, so the class will be given a data_path. To come up with the actual file paths, we concatenate data_path and our file name. For that reason, data_path should normally end with a slash (forward or back, depending on OS), to keep our data files in its own directory.

  The data store files should be deleted any time you want a completely fresh start. The data store files will also be wiped any time the syncrepl_client software version changes.

  Warning

  Data store files are also not compatible between Python 2 and Python 3. Attempting to use a data store from Python 2 with Python 3—or vice versa—will likely trigger an exception during instantiation.

• A callback class

  The callback class is an object (a class, or an instance). The callback class’ methods are called when the Syncrepl client receives updates.

  The complete list of callback methods is documented in BaseCallback. That class is designed for subclassing, because it defines each callback method but doesn’t actually do anything. You can have your class inherit from BaseCallback, and let it handle the callbacks that you don’t care about.

  For a simple example of a callback in action, see the code for the LoggingCallback class.

• An LDAP URL

  The LDAP URL contains all information about how the Syncrepl client should connect, what credentials should be used to connect, and how the search should be performed.

  The LDAPUrl class is used to parse the LDAP URL. You can also use ldapurl (part of the ldap-utils) to construct a URL. Refer to the class’ documentation for information on the fields available.

  If a valid data store exists, this field is optional: the URL your provide will be stored in the data store, which will be used in subsequent connections. If you provide both an LDAP URL and a valid data store, your LDAP URL will be used, as long as the search parameters have not changed (the LDAP host and authentication information are OK to change).

  syncrepl_client supports the following bind methods, which you control by using particular LDAP URL extensions:

  • Anonymous bind: Do not set a bind DN or password.

  • Simple bind: Set the bind DN and password as part of the URL.

    The bindname LDAP URL extension is used to hold the bind DN, and the X-BINDPW extension is used to hold the bind password.

    Note

    For security, it is suggested that you store the LDAP URL without a password, convert the URL into an ldapurl.LDAPUrl object at runtime, add the password, and pass the password-laden object to the Syncrepl constructor.

  • GSSAPI bind: Set the bind DN to GSSAPI, and do not set a password.

    Note

    You are responsible for ensuring that you have valid Kerberos credentials.

    As an extra safety mechanism, when you receive the bind_complete() callback, consider doing a “Who am I?” check against the LDAP server, to make sure the bind DN is what you expected. That will help guard against expired or unexpected credentials.

Methods are defined below. Almost all methods are documented, including internal methods.

Warning

Methods whose names start with syncrepl_ are internal methods, which clients must not call. That being said, the methods are still being documented here, for educational purposes.

__init__(data_path, callback, mode, ldap_url=None, starttls=False, **kwargs)

Instantiate, connect to an LDAP server, and bind.

Parameters:

• data_path (str) – A path to the database file.
• callback (object) – An object that receives callbacks.
• mode (A member of the SyncreplMode enumeration.) – The syncrepl search mode to use.
• ldap_url (str or ldapurl.LDAPUrl or None) – A complete LDAP URL string, or an LDAPUrl instance, or None.
• starttls (bool) – Weather STARTTLS should be used before binding.

Returns:

A Syncrepl instance.

Raises:

syncrepl_client.exceptions.VersionError,

syncrepl_client.exceptions.LDAPUrlError, syncrepl_client.exceptions.LDAPUrlConflict, sqlite3.OperationalError, ldap.SERVER_DOWN

This is the Syncrepl class’s constructor. In addition to basic initialization, it is also responsible for making the initial connection to the LDAP server, binding, and starting the syncrepl search.

Note

Many parts of this documentation refers to syncrepl as a “search”. That is because a syncrepl is initiated using an LDAP search operation, to which a syncrepl control is attached.

• data_path is used to specify the prefix for the path to data storage. Syncrepl will open multiple files, whose names will be appended to data_path. You are responsible for making sure that data_path is appropriate for your OS.

  Note

  Some basic checks may be performed on the data files. If you use a different version of software, those checks will fail, and the contents will be wiped.

• callback can be anything which can receive method calls, and which is specifically able to handle the calls defined in BaseCallback.

• mode should be one of the values from SyncreplMode. REFRESH_ONLY means that you want the syncrepl search to end once your have been brought in sync with the LDAP server. REFRESH_AND_PERSIST means that, after being refreshed, you will receive notice whenever a change is made on the LDAP server.

• ldap_url is an LDAP URL, containing at least the following information:

  • The LDAP protocol (ldap, ldaps, or ldapi).
  • The base DN to search, and the search scope.

  All other LDAP URL fields are recognized. The bindname LDAP URL extension may be used to specify a bind DN (or “GSSAPI” for GSSAPI bind). When using simple bind, the X-BINDPW extension must hold the bind password.

• starttls is a boolean value. If True, then after the initial connection is made to the LDAP server, STARTTLS will be used to secure the connection. Binding will only take place when the STARTTLS operation is completed.

  Note

  STARTTLS is only usable when using the ldap connection scheme.

The bind_complete() callback will be called at some point during the constructor’s execution.

Returns a ready-to-use instance. The next call you should make to the instance is poll(). Continue calling poll() until it returns False; then you should call unbind(). To request safe teardown of the connection, call please_stop().

unbind()

Safely save state and disconnect from the LDAP server.

Returns:None

If you have instantiated this object on your own, call unbind to ensure that all data files are flushed to disk, and the LDAP server connection is properly closed.

Warning

If you are using the Context Manager protocol, do not call `unbind`; it will be called for you at the appropriate time.

Warning

Not all Python implementations delete objects at the same point in their code. PyPy, in particular, is very different. Do not rely on assumptions about garbage collection!

Once unbound, this instance is no longer usable, even if it hasn’t been deleted yet. To start a new client, make a new instance of Syncrepl.

db()

Return a sqlite3 database instance for client use.

Returns:A DBInterface instance.

Returns an instance of DBInterface, which you can use.

Note

The instance returned will be bound to the local thread.

Warning

Please read, understand, and observe all of the notes and warnings in the DBInterface class!

db_reconnect()

Close and reopen the database connection.

Returns:None.

This method closes and reopens the database connection. It is meant to be called if this instance is transferred a new thread, because the database connection is not thread-safe.

Note

Calling this method does not affect database connection instances returned from the db() method. However, the warning about thread-safety still applies: Instances returned by db() are not thread-safe.

Warning

All uncommitted transactions will be rolled back when this method runs, and all cursors (such as those obtained from callbacks) will become invalid (although you shouldn’t be using them outside of a callback, so it should not matter).

please_stop()

Requests the safe termination of a Syncrepl search.

Returns:None.

After calling this method, there is a set list of steps your code should take:

1. Continue calling poll() until it returns False.
2. Call unbind() (unless you’re using the Context Management protocol).
3. Stop using this instance.

When running in refresh-only mode, this does nothing: Interrupting a refresh is dangerous, because there is no guarantee that the updates from the LDAP server are being received in any particular order. The refresh will be allowed to complete, and then it is safe to stop.

When running in refresh-and-persist mode, if the refresh phase is still in progress, it will be completed. If in the persist phase, a Cancel request will be sent to the LDAP server. Operations will then continue until the LDAP server confirms the search is cancelled.

This is the only method which is thread-safe.

poll()

Poll the LDAP server for changes.

Returns:True or False.

In refresh-only mode, returning True indicates that the refresh is still in progress. You must continue calling poll() until False is returned. Once False is returned, the refresh is complete, and it is safe to call unbind().

In refresh-and-persist mode, returning True only indicates that the connection is still active: Work might or might not be taking place. The refresh_done() callback is used to indicate the completion of the refresh phase and the start of the persist phase. During the refresh phase, when the connection is idle, poll() will return True every ~3 seconds. This is for single-process applications.

Most callbacks will be made during the execution of poll().

Warning

Just because poll() has returned, does not mean that you are in sync with the LDAP server. You must continue calling poll() until it returns False.

To request safe, consistent teardown of the connection, call please_stop().

run()

Run poll() until it returns False.

Returns:None

Runs the poll() method continuously, until it returns False. This is a callable.

In refresh-only mode, this method is good to use, as it saves you from having to write a while loop. Once this method returns, you know that the refresh has completed, and you are clear to call unbind() to clean up the instance.

In refresh-and-persist mode, this method should only be called when you are running this instance in its own thread. It will call poll() effectively forever, until the LDAP server goes away. For that reason, you should use this method as the target to pass to the threading.Thread constructor. Doing so allows the Syncrepl search to run while your main thread can get on with other work. In particular, your main thread should catch signals like SIGHUP, SIGINT, and SIGTERM.

Note

When a Syncrepl search is actively running, most of the execution time is spent inside OpenLDAP client code, waiting for updates from the LDAP server. If OpenLDAP client code receives a signal, it normally responds by abruptly closing the LDAP connection and raising an exception. That will cause the Syncrepl search to stop in an unsafe manner.

You really should run refresh-and-persist Syncrepl searches in a thread.

Before starting to call poll(), this method will call db_reconnect() to open a database connection that is tied to this thread (although it is _not_ thread-local).

When you are running this method in a thread, use please_stop() to request the safe shutdown of the Syncrepl search. Once the thread has been join()-ed, remember to call unbind() to clean up the instance.

Warning

please_stop() is the only thread-safe method! Once you have spawned your Syncrepl search thread, no other methods (except for please_stop()) may be called until the thread has been join()-ed.

syncrepl_get_cookie()

Get Syncrepl cookie from data store.

Returns:bytes or None.

Note

This is an internal Syncrepl operation. It is documented here for educational purposes, but should not be called by clients.

Called at the start of the syncrepl operation, to load a cookie from state. If present and valid, the LDAP server will know how far behind we are.

If not present, or invalid (typically because it’s too old), the LDAP server will start us over, as if we were a new client.

syncrepl_set_cookie(cookie)

Store Syncrepl cookie in data store.

Parameters:cookie (bytes) – An opaque string. Returns:None.

Note

This is an internal Syncrepl operation. It is documented here for educational purposes, but should not be called by clients.

Called regularly during syncrepl operations, to store a “cookie” from the LDAP server. This cookie is presented to the LDAP server when we reconnect, so that it knows how far behind we are.

syncrepl_refreshdone()

Mark the transition from the refresh phase to the persist phase.

Returns:None.

Note

This is an internal Syncrepl operation. It is documented here for educational purposes, but should not be called by clients.

This is called when we moving from the Refresh mode into the Persist mode of refresh-and-persist. This is not called in refresh-only mode.

Triggers a refresh_done() callback.

syncrepl_delete(uuids)

Report deletion of an LDAP entry.

Parameters:uuids (List of binary.) – List of UUIDs to delete. Returns:None. Raises:syncrepl_client.exceptions.DBConsistencyWarning

Note

This is an internal Syncrepl operation. It is documented here for educational purposes, but should not be called by clients.

Called when one or more entries have been deleted, or have changed such that they are no longer in our search results.

The one parameter is a list of UUIDs, which we should already know about.

Triggers a record_delete() callback for each UUID.

syncrepl_present(uuids, refreshDeletes=False)

Indicate the presence or absence of an LDAP entry.

Parameters:

• uuids (List of bytes, or None.) – List of UUIDs present or absent.
• refreshDeletes (boolean) – Indicates presence or absence.

Returns:

None.

Note

This is an internal Syncrepl operation. It is documented here for educational purposes, but should not be called by clients.

This function is used in refresh-only syncrepl, and in the refresh phase of refresh-and-persist syncrepl. It is not used in the persist phase.

As part of the syncrepl process, we get a big list of UUIDs and their DNs (plus attributes), from which we build a mapping (see syncrepl_entry(), below). The first time a sync takes place (when there is no valid cookie), you are able to assume that every mapping entry received is present in the directory; but in subsequent syncs (using a valid cookie) you can’t be sure which entries are present and which have been deleted. In addition, if you have a cookie that is now too old, there is no way to know which entries in your data store still exist in the directory.

The “Present” messages, and the resulting calls, are used to bring us back in sync with the Directory, regardless of our local state.

uuids is either a list of UUIDs, or None. refreshDeletes is a boolean. To understand how the two parameters are related, it’s better to look at the latter parameter first.

• If refreshDeletes is False, and uuids is a list, then uuids contains a list of entries that are currently in the directory.
• If refreshDeletes is False, but uuids is None, then we are almost synced. We now need to go into our mapping, and remove all entries that were not previously mentioned as being in the directory.
• If refreshDeletes is True, and we have a list, then uuids contains entries that used to be in the directory, but are now gone.
• If refreshDeletes is True, but uuids is None, then we are synced: Our current mapping of UUIDs, minus those previously deleted, represents the current state of the directory.

Here is another way to think about it: The LDAP server needs to work out the most efficient way of relaying changes to us. There are three ways of telling us what has changed:

• “The following entries are in the directory; everything else you knew about has been deleted.”

  This is the easiest way of informing us of changes and deletions.

  In this mode, you will receive:

  • Calls where uuids is a list and refreshDeletes is False.
  • A call where uuids is None and refreshDeletes is False.

• “The following entries are new, and these other entries have been deleted, but everything else you know about is still in the directory.”

  This is the mode that is used when, since your last checkin, there have been alot of additions and deletions.

  In this mode, you will receive:

  • Calls where uuids is a list and refreshDeletes is False.
  • Calls where uuids is a list and refreshDeletes is True.
  • A call where uuids` is None and refreshDeletes is True.

• “Everything is up-to-date and there are no changes.”

  When things are quiet, this is the mode that is used.

  In this mode, you wil receive:

  • A call where uuids is None and refreshDeletes is True.

The LDAP server chooses which mode to use when we connect and present a valid cookie. If we don’t have a valid cookie, then the LDAP server falls back to mode A.

syncrepl_entry(dn, attrs, uuid)

Report addition or modification of an LDAP entry.

Parameters:

• dn (str) – The DN of the entry.
• attrs (Dict of List of bytes.) – The entry’s attributes.
• uuid (bytes) – The entry’s UUID.

Returns:

None.

Note

This is an internal Syncrepl operation. It is documented here for educational purposes, but should not be called by clients.

This function is called to add entries to a map of UUIDs to DN/attribute-list pairs. It is also used to change an existing DN: In that case, the UUID matches an existing entry, but the DN is different.

DNs are not static - they can change. That’s a problem when you are trying to track changes over time. To deal with that, the LDAP server assigns each entry a UUID. We then maintain a mapping of UUIDs to DNs, because all future syncrepl-related calls will refernce UUIDs instead of DNs.

In refresh-only sync, and in the refresh phase of refresh-and-persist syncrepl, this method is called multiple times, interspersed with calls to syncrepl_present(). If a valid cookie was provided, the server will only send new/changed entries since our last checkin; otherwise, we’ll get a big list of entries—all of which will be present—to seed our mapping.

In refresh-and-persist mode, everything from the previous paragraph is true, but when in the persist phase (once the refresh phase has completed) we should expect to be called at random times as the server sends us updates to our mapping.

The set of attributes is the intersection of three sets:

• The populated attributes of a particular entry.
• The attributes you are allowed to see.
• The attributes you requested in your search.

All attribute entries are lists of binary strings. Lists are used throughout because the LDAP client does not know which attributes are multi-valued, and binary strings are used because the LDAP client does not know each attribute’s syntax. The client is responsible for knowing the directory’s schema, and casting/converting values appropriately.

class syncrepl_client.SyncreplMode

This enumeration is used to specify the operating mode for the Syncrepl client. Once a mode is set it can not be changed. To change the mode, you will have to (safely) shut down your existing search, unbind and destroy the existing instance, and start a new instance in the new mode.

REFRESH_ONLY = 'refreshOnly'

In this mode, the syncrepl search will last long enough to bring you in sync with the server. Once you are in sync, poll() will return False.

REFRESH_AND_PERSIST = 'refreshAndPersist'

In this mode, you start out doing a refresh. Once the refresh is complete, subsequent calls to poll() will be used to receive changes as they happen on the LDAP server. All calls to poll() will return True, unless a timeout takes place (that will throw ldap.TIMEOUT), you cancel the search (that will throw ldap.CANCELLED), or something else goes wrong.

Note

When running a Syncrepl search in refresh-and-persist mode, it is strongly recommended that you run the actual search operation in a thread, so that you can catch signals which would otherwise cause an unclean termination of the Syncrepl search.

For more information, see the run() method, which is what you should use as the thread’s target.