Raising Events with Files

File events are special events that are linked to a directory on the host that a platform agent runs on. If they are defined the platform agent will monitor the file system and wait for a file to arrive that matches the file pattern. File events are supported by all UNIX, Microsoft Windows and HP OpenVMS platform agents.

Several strategies are available to efficiently and reliably detect files.

A reliable detection will detect file changes exactly once so not too often, but without missing any changes either. Detection should not be resource intensive.

Furthermore, when a process then looks for the file it should be present - in other words the file system detection event should not detect file changes that are then not present when a process runs and checks the file.

The most efficient and reliable way to detect file changes and convert these into unique file events is to detect local files only, with a move directory so that all files result in exactly one event, and with file age checks to prevent expensive lock tests.

FileEventCredential Process Server Parameter

The FileEventCredential process server parameter allows you to specify a user that will be used to detect files and move files, if a move directory was specified.

Syntax

[<NTLM DOMAIN>\]<real_user>@<domain> - uses credential `login-<real_user>-<domain>`
<NTLM DOMAIN>\<user>/<password> - hardcoded password (not recommended)
<real_user>/<password>@<domain> - hardcoded password (not recommended)

• <NTLM DOMAIN> - Legacy NTLM domain name, such as MYDOMAIN.
• <real_user> - Windows username as specified in the Real User of the credential, such as jdoe.
• <domain> - ActiveDirectory domain, also known as Kerberos realm or DNS domain name, such as mydomain.example.local.

Networked file systems

It is highly recommended that you only use file events on local files, not networked files on a NFS or SMB share. This is because the remote file system will always introduce some unreliability, and this may cause spurious or missed file events. If you detect files on networked file systems make sure that the jobs that depend on the file system are reliable in that they do not cause errors when a file is suddenly missing because of caching effects or file server reboots.

If you want networked files to be detected on Windows, you must specify a valid credential in the FileEventCredential process server parameter. Note that the user defined in the credential must at least have permissions to read the file and if a move directory is involved, the user must have permissions to move it (this means read and delete on the source file and create in the target directory). The process server parameter must be set because the platform agent is running as NT Authority\\System; this account has no access to network shares/mapped drives.

Networked file events are not supported on UNIX/HP OpenVMS.

Files located on a SAN are considered local files. NFS or SMB mounted files (NAS) are considered networked files.

File events are provided by the platform agent, which requires a ProcessServerService.OS.limit license key for UNIX and Microsoft Windows agents, and a ProcessServerService.VMS.limit license key for OpenVMS agents.

The file event definition sets the:

• Directory where the file(s) are detected.
• Filename or pattern that the file must have.
• Minimum file size that the file must have.
• Minimum age that the file must have since last modification.
• Lock flag; if set then the system checks that the file is not locked or open by any process.
• An optional move directory further ensures that files are detected only exactly once.
• Overwrite flag; if set then detected files will overwrite existing files in the move directory.
• Poll interval; how often the file event is checked.
• Raise comment; allows a custom comment to accompany the raised event.

Detection

If you do not specify a move directory the files are left in place once detected. If their timestamp is changed or they are deleted and then recreated they will be detected again. If you specify a wildcard an event will be raised for every file that changes or is created. To make this work reliably the platform agent keeps a state file that contains the timestamp for every file.

The HP OpenVMS file system supports versioning of files. To ensure reliable detection all versions of a file will be detected, even if you specify a specific version in the filename attribute.

Wildcards

You cannot have wildcard characters in the directory path, but you can have wildcards in the filename.

A filename is a wildcard pattern if it contains one of the characters ?, * or [. Globbing is the operation that expands a wildcard pattern into the list of pathnames matching the pattern. Matching is defined by:

• A ? character matches any single character.
• A * matches any string, including the empty string.

Character classes

An expression [...] where the first character after the leading [ is not an! matches a single character, namely any of the characters enclosed by the brackets. The string enclosed by the brackets cannot be empty; therefore ] can be allowed between the brackets, provided that it is the first character. (Thus, [][!] matches the three characters ], [ and!)

Ranges

There is one special convention inside a range: two characters separated by - denote a range. (Thus, [A-Fa-f0-9] is equivalent to [ABCDEFabcdef0123456789])

One may include - in its literal meaning by making it the first or last character between the brackets. (Thus, [*--] matches the four characters *, +, ``,, -)

Complementation

An expression [!...] matches a single character, namely any character that is not matched by the expression obtained by removing the first! from it. (Thus, [!]ab] matches any single character except ], a and b)

One can remove the special meaning of ?, * and [ by preceding them by a backslash.

Between brackets these characters stand for themselves. Thus, [[?*\\] matches any of the four characters [, ?, * or \.

Subdirectories

If you want to detect files in subdirectories you can use */<pattern> to match any subdirectory at the first level or **/<pattern> to match any level of directory. You can use <pattern1>/<pattern2> to use glob style matching for both the directory and filenames.

note

As of version 9.0.6 the agent uses its own 'glob' parser. If you need the old platform-specific parser as used in version 9.0.5 and before you can set JCS_OLD_GLOB_STYLE to any value in the agent environment.

Minimum Age

The minimum age is a time-based method to ensure the file is complete.

• You set the minimum age to a positive integer and the agent will wait the specified value in seconds from the last modification time.
• You set the minimum age to a negative integer and the agent will ensure the timestamp of the file does not change for the specified absolute value in seconds, disregarding the last modification time offset.

The field accepts values in the range ( -86400 to 86400, +/-24 hours).

Example:

The platform agent is running on a system where the current time is 9:30 AM, a file arrives with a timestamp of 11:30 AM.

1. If the Minimum Age field is set to 60, the event will be fired at 11:31 AM.
2. If the Minimum Age field is set to -60, the event will be fired at 9:31 AM.

Another example:

The platform agent is running on a system where the current time is 9:30 AM, a file arrives with a timestamp of 9:30 AM.

1. If the Minimum Age field is set to 60, the event will be fired at 9:31 AM.
2. If the Minimum Age field is set to -60, the event will be fired at 9:31 AM.

Note that on UNIX systems it is highly recommended to use a non-zero minimum age instead of a lock test as it is much more efficient use of computing resources.

Lock detection

The platform agent can check for locks on the file when the lock attribute is set.

On Microsoft Windows and HP OpenVMS this is done by checking that there are no outstanding file locks. This works quickly and reliably on local files.

On UNIX checking for locks would not work as UNIX does not lock files that are open for read or write; therefore we use the "fuser" or equivalent program to check in the kernel whether any process has the inode of the file open. This has two potential issues: performance and reliability.

The performance of the lock test program depends on how the UNIX system stores open file descriptors. Some operating systems must enumerate all open files and take several seconds to find out whether a file is locked.

Also, if the file is stored on an NFS file system the lock test will not reveal any locks held by other systems.

Instead of the lock test we suggest using the minimal file size and minimal file age test.

Changing the lock test on UNIX

You can set the command used to perform the file-is-locked check for UNIX platform agents using the FileInUseCommand process server parameter. Its default value is:

Operating System	Command
Linux	lsof "${Filename}"
UNIX	/usr/sbin/fuser "${Filename}"
OpenVMS	N/A
Microsoft Windows	N/A

You can use your own command but it should behave like either fuser or lsof. The rules that the agent uses to parse the output are:

• For fuser, if it finds at least one line of the format ^${file}:[ ]*[0-9]+ then the file is locked. Note that the fuser implementation generally either is silent on an unlocked file (for example on Linux) or it produces (for instance on AIX 5): ^${file}:[ ]* so the test on digits after the whitespace is necessary.
• For lsof, if the result value is error and the output is empty, the file is not locked. If the result value is ok (and there is output) the file is locked.

The system determines whether to check the lsof type output by scanning the FileInUseCommand for the string lsof.

Move Directory

If your wildcard matches many files it is more efficient to use a move directory. This allows the system to move detected files to a new directory, keeping the directory that it needs to watch smaller and thus more efficient. This is also helpful at the OS level: if a file is still in the original directory it has not been detected yet; if it is in the move directory the system has seen the file and the event has been raised (or is in the process of being raised). If the filenames that are detected are not unique over time you should set the overwrite flag, or delete the detected file as part of the actions that happen as part of handling the event. By default the system will not overwrite the file, as the content may be needed by the system. If your files contain data you should keep the overwrite file checkbox cleared; the platform agent will then ensure that no data is lost.

The move directory field value can be:

1. A full path including a directory and a filename.
2. Just a directory name, in which case the filename will remain the same.
3. Just a file name in which case the file will remain in the same directory with the new filename. It is your responsibility to ensure that the new filename will not match the file event (or else it will be detected with the new name in the next poll.) Using this option is not as efficient as a move directory that contains a directory.

The directory path that the move directory contains can be a subdirectory of the original directory (unless the filename specifies subdirectories.)

In order to be efficient and reliable the move directory should be on the same file system as the detection directory. Do not use a move directory on a different file system, even if this is not explicitly forbidden by the system. Some operating systems, on some file systems, transparently convert a move operation into a copy and delete operations. This will result in occasional failures and is not recommended.

note

The directory path set as the move directory must exist.

The HP OpenVMS file system supports versioning of files. To ensure that no files are clobbered the file move performed by the agent will not modify the file version of the moved file, and refrain from creating a new file version. If you place a file in the detection directory with a version number that already exists in the move directory the file will not be moved and an error is raised unless you have set the Overwrite flag.

File Event Move Directory

All DateTime variables use the format yyyyMMddHHmmss; you can specify a JavaDateFormat using the ${FileDateTime:HHmmss} syntax.

• ${BaseDirectory} - path to the directory containing the detected file.
• ${BaseName} - base filename of the detected file.
• ${Name} - filename of the detected file.
• ${Dot} - dot (``.) such as found in filenames before the extension.
• ${Extension} - extension of the detected file.
• ${CurrentDateTime} - agent date & time when the file was detected.
• ${CurrentTimeStamp} - agent timestamp at which the file was detected, in a numeric format containing the milliseconds since 1970, usually referred to as epoch or UNIX time.
• ${DateTime} - deprecated in favor of CurrentDateTime.
• ${TimeStamp} - deprecated in favor of CurrentTimeStamp.
• ${FileDateTime} - File modification date & time.
• ${FileTimeStamp} - File modification time in epoch.
• ${fileName} - filename (full path of file) of the detected file.
• ${UniqueId} - the unique ID of the file event.

note

${DateTime} and ${TimeStamp} have been deprecated and should not be used anymore, the equivalents are ${CurrentDateTime} and ${CurrentTimeStamp}, respectively.

Poll interval

The poll interval defines how many seconds elapse between checks for this file event.

Raiser Comments

File Event Raiser Comments

Events can also be raised by files on servers with a platform agent or on AS/400 systems.

The following substitution parameters are available for raiser comments of file events:

All DateTime variables use the format yyyyMMddHHmmss; you can specify a JavaDateFormat using the ${FileDateTime:HHmmss} syntax.

• ${CurrentDateTime} - agent date & time when the file was detected.
• ${CurrentTimeStamp} - agent timestamp at which the file was detected, in a numeric format containing the milliseconds since 1970, usually referred to as epoch or UNIX time.
• ${FileDateTime} - File modification date & time.
• ${FileTimeStamp} - File modification time in epoch.
• ${ServerDateTime} - Central server date & time when the event was raised.
• ${ServerTimeStamp} - Central server time in epoch when the event was raised.
• ${processServer - name of the process server.
• ${server} - Platform agent name.
• ${filename} - path of the detected file (before any move).
• ${finalPath} - new path of the detected file (after any move).

note

${DateTime} and ${TimeStamp} have been deprecated and should not be used anymore, the equivalents are ${CurrentDateTime} and ${CurrentTimeStamp}, respectively.

The default file event raiser comment is the following:

File event raised by "${filename}" on "${server}"

The following topics cover file events in more details:

• Triggering Events with Files
• Files on AS/400 Trigger Events

Error handling

When a platform agent encounters a problem with a file that it has detected it will take the following actions:

• A message is logged in the network processor log file.
• An operator message attached to the file event definition is sent to the server.
• The file event scan interval is set to a large value. At the first error it is set to one hour. If the value is already one hour or longer it is doubled until the scan interval is once per day. If the next attempt succeeds the scan interval is reset to the original value.

The scan interval is increased as repeating the attempt after the original scan interval would generate huge amounts of error messages.

If you see file event error messages in the operator message console and have repaired the problem then you should reset the platform agent event timing by stopping and then starting the process server. The error messages that cause this are accompanied by a message stating until when the platform agent will refrain from checking the files for this file event. The current list of possible errors includes:

1. Could not check whether file is in use: {command}.
2. File event {name} not raised because the file could not be renamed from {source} to {destination}.
3. File event {name} not raised for detected file {source} because {directory} is not an existing directory.
4. File event {name} not raised for detected file {source} because the variables in {moveDirectory} could not be substituted.
5. File event {name} not raised as the target move file name could not be built: Components {moveDirectory} and {targetFile}.

note

Different file events each have an individual scan interval; when one file event definition errors on a particular system then it does not cause any other file events defined on the same process server to delay.

Procedure

1. Choose "Definitions > Event Definitions".
2. Choose Refresh to see all existing event definitions.
3. Choose New to create an event definition.
4. Fill in the details, refer to the section below.
5. Choose the File Event Definition tab and choose the New button.
6. Fill in the details, refer to the section below.
7. Choose Save and Close to save the event definition.

Event Definition

Tab Name	Field Name	Description
Event Definition	Name	The name of the event definition. Names cannot start with a number, can contain only letters, numbers, and underscores (_). Limited to 80 characters.
Event Definition	Application	The name of the application used to group this object.
Event Definition	Description	An optional description of the event definition. Free-form text, any printable UTF-8 character combination is allowed; note that depending on the platform/browser, some characters of the symbol and other Unicode tables will not be displayable. Limited to 255 characters.
Event Definition	Requires Raiser Comment	Does this event definition require a comment when it gets raised?
Event Definition	Raiser Comment	Comment when it gets raised
Event Definition	Requires Clearer Comment	Does this event definition require a comment when it gets cleared?
Event Definition	Clearer Comment	Comment when it gets cleared
Documentation	Documentation	An optional comment about the event definition. Free-form text, any printable UTF-8 character combination is allowed. No limit on the number of characters, however, it is recommended to use Documents if you want to use more than 255 characters and link to them from the Documentation field.
File Event Definition	Name	The name of the file event definition. Free-form text, any UTF-8 printable character combination is allowed. Limited to 80 characters.
File Event Definition	Description	An optional description of the file event definition. Free-form text, any printable UTF-8 character combination is allowed; note that depending on the platform/browser, some characters of the symbol and other Unicode tables will not be displayable. Limited to 255 characters.
File Event Definition	Comment	An optional comment about the file event definition. Free-form text, any printable UTF-8 character combination is allowed. No limit on the number of characters, however, it is recommended to use Documents if you want to use more than 255 characters and link to them from the Documentation field.
File Event Definition	Enabled	Is this file event enabled? Disabled file events will not trigger events.
File Event Definition	Process Server	The process server which runs on the server.
File Event Definition	Directory	Directory to await the incoming file
File Event Definition	Pattern	Pattern of the file name, can be explicit file name or contain GLOB (* and ?) wildcards
File Event Definition	Move Directory	Once the file has been picked up, move the file (use of this is recommended at it has less overhead)
File Event Definition	Overwrite file	Whether any existing files in the Move Directory will be overwritten
File Event Definition	Poll Interval	Interval to poll the file system.
File Event Definition	Minimum Size	The minimum file size in bytes; the event will not be raised until the minimal size has been reached.
File Event Definition	Minimum Age	The minimal age in seconds since the file modification; the event will not be raised until the minimal age has been reached. Default is 0, this means that files that were modified in the future are not detected by default until the last modification time has been reached. This property is recommended for files residing on NFS shares, since file locking is not reliable. Negative numbers force the agent to disregard the time offset of the last modification time; you use this when the file was generated on a system that does not have its clock in sync with the one where the file event fires.
File Event Definition	Check Lock	Check the file lock, the process server will wait until the lock on the file has been released.
File Event Definition	Raise Comment	Comment that will accompany the raised event, can contain substitution variables.

Example

1. Choose "Definitions > Event Definitions" in the Navigation bar.
2. Choose the New button and fill in the values for the Event Definition Tab.
3. On the File Event Definition tab choose the New button and fill in the values below.

Event Definition Tab

Name: DataWareHouseLoaded
Description: Data Warehouse loaded
Comment: The data warehouse has been loaded. Reports that require data from the data warehouse should wait for this event to be raised.

File Event Definition Tab

Name: DataWareHouseLoaded
Description: Data Warehouse loaded
Comment: The data warehouse has been loaded. Reports that require data from the data warehouse should wait for this event to be raised.
Enabled: <true>
Directory: C:\tmp\
Pattern: done??.txt
Move Directory: C:\done\
Check Lock: <true>
Process Server: WIN-SERV-03