How do people implement ingesting the ondemand logs?
From the help, we require:
xdmod-ondemand-ingestor -r -u -d
Resource is straightforward.
From using the --dry-run flag, I’ve deduced that
- can just be essentially a (short-hand or fqdn) name for the ondemand server
- needs to be a local directory, i’m trying /tmp
Result of ‘xdmod-ondemand-ingestor -r ondemand -u ondemand01 -d /tmp’
was to just create the modw_ondemand data structures, but not to ‘ingest’ any data from the ondemand server.
I’m sure that i’m missing crucial details. I’m hoping to pull together some notes to offer for enhancing the existing documentation around the xdmod/ondemand integration.
Thanks
Documentation seems to indicate -d is the input to the program, whereas you seem to be using it as the output location? I think it’s meant to be where your apache logs are - likely /var/log/httpd.
Yeah - a bit confusing. xdmod-* commands run on the xdmod server. The ‘-d’ is required for ‘xdmod-ondemand-ingestor’, so the question of what people have been using for ingesting the ondemand logs remains open.
I thought the -u worked together with -d , and so indicated /var/log/ondemand-nginx, and the result was ‘invalid path’ (approximately, can’t remember details).
Time will tell. Thanks for thinking about it, Jeff
xdmod-ondemand-ingestor reads log files from the filesystem on the Open XDMoD web server. If this is a different web server than the Open OnDemand web server, then you’ll need to set up an intermediate process to copy the log files over before they are ingested. The -u option was removed in Open XDMoD 11.0. In prior versions, the -u option wasn’t used for transferring data; it was used for matching the value of the Referer header in the logs files, and only matching lines would be ingested. I will note that Open XDMoD 11.0 made some substantial changes/fixes to how Open OnDemand data are ingested and categorized; these are detailed in the 11.0 upgrade guide.
Aaron – Thanks very much. I misunderstood the documentation (still working with v10.5, as you can tell).
To be clear, are we talking about copying /var/log/ondemand-nginx// paths from ondemand server to xdmod server?
And whilest I continue to struggle with authentication, should I abandon v10.5, and upgrade to v11.0, and start again with the authentication?
Thanks
xdmod-ondemand-ingestor ingests Apache access logs (e.g., from /var/log/httpd/).
I would recommend backing up your work on the authentication and then upgrading to Open XDMoD 11.0. The upgrade guide is here.