Completed

Log collector shell script

The assignment is to create a log collector shell script that reads csv-formatted log events from stdin and publishes them in json format to a remote server via HTTP requests.

The functional requirements are as follows:

-The script is launched by a daemon process which constantly writes event records to stdin of the script.

-The script buffers records until either the buffer size exceeds x records or x seconds has elapsed since first record in the buffer.

-When the buffer is flushed, the records are sent to remote server via HTTP POST request (using curl)

-Input is in csv format (i.e value1,value2,value3 ...). Some fields are transformed using lookup maps and then mapped to a json record (i.e {"field1": "transformedValue1", "field2": "value2", "field3": "transformedValue3"...}). Payload of the POST request is a json array of the records.

-If record mapping fails (i.e field value not found in lookup map when transforming a value), the record is ignored.

-If flushing fails due to network error, the script should keep retrying until it succeeds.

-If flushing fails due to request error (HTTP status other than 200), the script should write error reason to stderr and exit.

-The script should not continue reading from stdin until the buffer is flushed succesfully

-The script accepts remote server address and an authentication token (passed as a header with the POST requests) via command line arguments. Other configurations are defined as constants in the beginning of the script.

Skills: Shell Script

See more: shell script remove line beginning, shell script delete lines beginning file, shell script monitor oracle application server log, aws linux logs, linux log analyzer, ecs logs to cloudwatch, ecs log collector, logs from ecs task, ecr logs, ecs logs, ecs log location, unix shell script alert log file, linux shell script log parsing, shell script delete log, ldap log rotation using shell script, shell script add line beginning file, shell script scan log files, use shell script create csv file log file, shell script remove lines beginning, shell script update write log file results

About the Employer:
( 0 reviews ) Tallinn, Estonia

Project ID: #17867449

Awarded to:

kaloyan13

I am master in Bash, and have worked with everything you mentioned in your description - csv, json, POST-ing, etc.

€60 EUR in 1 day
(146 Reviews)
6.2

9 freelancers are bidding on average €140 for this job

romanapanovich

hi Can i develope it in php-cgi? It is simple - I can use #!/usr/bin/php or something similar in the start of script and chmod 7 to file - and it will work as shell script - it will be simple for me about some ideas More

€170 EUR in 2 days
(1 Review)
3.8
VarunUnix

Hello, Not sure if I understood your requirement fully. The base requirement is clear like you need to collect data from a csv and push into a remote http server. That can be easily done - by the way do you have endpo More

€111 EUR in 4 days
(10 Reviews)
3.2
tangramua

Hello! I can help you to develop a log collector shell script according to your requirements. As for me I'm a professional system admin with 10+ years of experience. Check my reviews. Recently I've successfully c More

€155 EUR in 3 days
(1 Review)
2.9
adampohp79

Hello, sir Your job is exciting me. Let me do this job for you now just to prove my expertise in this field. Hope I could awake you interest in me. I am always online, work hard and very sensitive to details. Thank More

€155 EUR in 3 days
(0 Reviews)
0.0
deytps86

Hello I am working in scripting technologies for 20 years. I worked in various shells c, korn, bash in various Unix system. Can you please share the URL and sample log file to parse and output format etc? Thanks!

€220 EUR in 3 days
(0 Reviews)
1.8
kselax

I can do it .r

€155 EUR in 3 days
(0 Reviews)
0.0
Fabricio83it

Knowledge to do the magic simply happens how you waaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaant!

€100 EUR in 4 days
(0 Reviews)
0.0
meysammahfouzi

Hi I am a senior C++ developer who has been programming on Linux systems for more than 8 years. I have been dealing with shell and python scripts during the same period of time because of the nature of my job. When we More

€133 EUR in 2 days
(0 Reviews)
0.0