YaDDNet Data Backup - Make It Tea

Try to learn something about everything, and everything about somethingThomas Huxley “Darwin's bulldog” (1824-1895)

Trace: • Raspberry Pi Farm • Code & Command Snippets • Monobook template wikis • Yaesu FT-891 • Yaesu FT-2980 • Radio Topics • YaDDNet : Running Costs • YaDDNet : New VPS • Icom IC-7200 • YaDDNet : VPS SSL Renewal

You are here: Home » Public Area » Radio » Radio Topics » YaDDNet » YaDDNet Data Backup

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.

<-[[.:start]]


====== YaDDNet Data Backup ======

** Offsite Text file backups of all DSC messages **

  * Alan Spindel initially suggested an offsite backup of the DSC messages in 2019
  * YaDDNet has had the facility to send copies of each new message out to specified IP addresses via UDP since that time
  * The DSC messages are sent by UDP as simple plain text, for easy archiving
  * The archived text file can be used if necessary to rebuild the SQL database table containing the DSC messages in case of disaster
  * Since I'm now hosting YaDDNet again I decided I should run a UDP log receiver at home to improve the backup provision
  * The daily archive files will be ''gzipped'' and stored on Dropbox
    * I could share the link to anyone who is interested in using the saved data

===== YaDDNet UDP Send to Mirror =====

  * In the ''PyYadd.py'' function a DSC message is created from the raw incoming data from YaDD (and DSC Decoder) in a standard format.
  * This involves converting MMSI -> Coast or Ship name, finding the ''[MID]'' and corresponding country etc.
  * Prior to using the newly created message to inject new data into the SQL server ''PyYadd.py'' calls the function ''send_to_mirror()'' to send out the UDP packet to any configured IP addresses
  * ''PyYadd.py'' then inserts the new data in the SQL server 
  * Any addressed & listening UDP clients receive the UDP packets and append them to a log file

<code python>

def send_to_mirror(data):
    log_data = "[log];"+data
    port = 2505
    host = "aaa.bbb.ccc.ddd.eee"
    host2 = "fff.ggg.hhh.iii"
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.sendto(log_data, (host, port))
    sock.sendto(log_data, (host2, port))

    return
</code>
===== Backup UDP receive and logrotate =====


==== UDP Listener ====

<code python>

#!/usr/bin/env python
#
# udp_logger.py
#

import SocketServer
import socket
import threading

class ThreadedUDPRequestHandler(SocketServer.BaseRequestHandler):
   
    def handle(self):
        data = self.request[0].strip()
        socket = self.request[1]
        self.write_file(data)
       
    def write_file(self,text):
        filename = '/home/gm4slv/yaddlogs/yaddnet_udp_log.txt'
        f = open(filename, 'a+')
        log = text
        f.write(log+"\n")
        f.close()
        return
 
class ThreadedUDPServer(SocketServer.ThreadingMixIn, SocketServer.UDPServer):
    pass

if __name__ == "__main__":
    HOST, PORT = "", 2505
    server = ThreadedUDPServer((HOST, PORT), ThreadedUDPRequestHandler)
    ip, port = server.server_address
    server.serve_forever()
    # Start a thread with the server -- 
    # that thread will then start one
    # more thread for each request
    server_thread = threading.Thread(target=server.serve_forever)
    # Exit the server thread when the main thread terminates
    server_thread.daemon = True
    server_thread.start()
    server.shutdown()

</code>


  * The UDP Log Receiver listens to UDP/2505 for incoming packets from YaDDNet. These are of the form:

<code>

[log];2024-04-19 08:43:06;NLD_MB_HF1;2187.5;SEL;002050480;COAST;205;Belgium;Ostende Radio;SAF;636092799;SHIP;636;Liberia;CISNES, D5HF4, Cargo;TEST;NOINF;--;--;REQ;OK
[log];2024-04-19 08:43:28;BDXC-Han-Remote;12577.0;SEL;002241022;COAST;224;Spain;Coruna Radio;SAF;538005991;SHIP;538;Marshall Islands;NAVIG8 VIOLETTE, V7IH3, Tanker;TEST;NOINF;--;--;REQ;OK
</code>


  * each new packet is appended to the running log file ''/home/gm4slv/yaddlogs/yaddnet_udp_log.txt''
  * the file will grow indefinitely if left to itself!

==== Logrotate ===

  * run by Crontab every day at midnight
  * copies the current running log to a timestamped file
  * erases the contents of the running log file 
  * ''gzips'' the timestamped copy and moves it to //Dropbox//

<code bash>

## CRONTAB ENTRY

00 00 * * * /home/gm4slv/bin/yaddlog_rotate.sh /dev/null 2>&1
</code>

<code bash>

#!/bin/bash
#
# yaddlog_rotate.sh

DROPBOX=/home/gm4slv/Dropbox/Yaddlogs/
DIR=/home/gm4slv/yaddlogs

INFILE=yaddnet_udp_log.txt

TIMESTAMP=`date +%Y%m%d_%H%M`

OUTFILE=$TIMESTAMP\_$INFILE

cp $DIR/$INFILE $DIR/$OUTFILE
cat /dev/null > $DIR/$INFILE


gzip $DIR/$OUTFILE

mv $DIR/$OUTFILE.gz $DROPBOX

</code>

   * This creates a ''gzipped'' log file in my Dropbox of each day's new DSC messages

=== Starting the UDP Listener at boot-up ===

  * crontab entry to start it as a background job at ''@reboot''

<code bash>
#
# CRONTAB ENTRY
#
@reboot python /home/gm4slv/bin/udp_logger.py &

</code>


--- //John Pumford-Green 19/04/24 09:31 BST//

Last updated : ~~LASTMOD~~


===== Further Information =====

[[https://www.yaddnet.org/index.php | YaDDNet]]

{{tag>radio yaddnet}}

Navigation

Search

Toolbox