Log Analysis and Visulization

import utils
import os
from os import path
import datetime
import pandas as pd

Result

• Statistics Summary
  • transaction per second
  • time per block
  • size
  • gas

• Features vs Time / Block Height (with all shards / per shard)
  • staking transaction per second
  • total transaction per second
  • transaction per second
  • time per block
  • size
  • gas

if __name__ == "__main__":
    
    date = datetime.date.today()-datetime.timedelta(days=1)
    name = "mainnet_"+date.strftime("%Y_%m_%d")+".log"
    fig_dir = "../../graphs/test_logs/mainnet/" + date.strftime("%Y_%m_%d") + "/"
    log_dir = "/home/ubuntu/jupyter/monitor/mini_explorer/data/" + name
    html_dir = "../../docs/graphs/test_logs/mainnet/" + date.strftime("%Y_%m_%d") + "/"

    data = [html_dir, fig_dir]
    for directory in data:
        if not path.exists(directory):
            try:
                os.makedirs(directory)
            except:
                print("Could not make directory")
                exit(1)

    data = utils.read_data(log_dir)
    shard = utils.data_processing(data)
    
    utils.set_config({
        'ignore_printing_statistics_summary': False,
        'ignore_drawing_features_vs_time_per_shard' : True,
        'ignore_drawing_features_vs_block_per_shard' : True,
        'ignore_drawing_staking_features_vs_time_per_shard' : True,
        'ignore_drawing_staking_features_vs_block_per_shard' : True,
        'ignore_drawing_features_vs_time' : False,
        'ignore_drawing_features_vs_block' : True
    })
    
    config = utils.config
    ## get the index of the new chain started
    index = []
    for s in shard:
        i = s[s['time_per_block'] < 0].index.tolist()   
        if len(i) == 0:
            break
        i.insert(0,0)
        i.append(len(s))
        index.append(i)
    
    if len(index) == 0:
        new = pd.concat(shard)
        new.reset_index(inplace = True, drop = True)
        print("transaction for ", date, new['transactions'].sum())
        utils.visualization(new, fig_dir, html_dir, config)
    else:    
        for i in range(len(index[0])-1):
            df = []
            for s in range(len(shard)):
                df.append(shard[s].iloc[index[s][i]+1:index[s][i+1]])
            new = pd.concat(df)

            fig_path = fig_dir + "chain_" + str(i) + "/"
            if not os.path.exists(fig_path):
                os.makedirs(fig_path)

            utils.visualization(new, fig_path, html_dir, config)

transaction for  2020-04-25 167
Statistics summary for shard 0
==================================
Total data points: 10502
              size           gas  transaction_per_second  time_per_block
mean    755.113217  3.920690e+03                0.001738        8.225005
std     412.524383  1.032535e+05                0.015059        0.423043
min     738.000000  0.000000e+00                0.000000        8.000000
25%     738.000000  0.000000e+00                0.000000        8.000000
50%     738.000000  0.000000e+00                0.000000        8.000000
75%     738.000000  0.000000e+00                0.000000        8.000000
max   16039.000000  3.847731e+06                0.500000       12.000000

Statistics summary for shard 1
==================================
Total data points: 10572
              size           gas  transaction_per_second  time_per_block
mean    745.069618      7.945516                0.000045        8.171349
std     700.800998    408.421954                0.002300        0.376800
min     738.000000      0.000000                0.000000        8.000000
25%     738.000000      0.000000                0.000000        8.000000
50%     738.000000      0.000000                0.000000        8.000000
75%     738.000000      0.000000                0.000000        8.000000
max   72777.000000  21000.000000                0.125000        9.000000

Statistics summary for shard 2
==================================
Total data points: 10540
             size           gas  transaction_per_second  time_per_block
mean   738.194118     15.939279                0.000092        8.197343
std     11.364740    578.361940                0.003351        0.398013
min    738.000000      0.000000                0.000000        8.000000
25%    738.000000      0.000000                0.000000        8.000000
50%    738.000000      0.000000                0.000000        8.000000
75%    738.000000      0.000000                0.000000        8.000000
max   1858.000000  21000.000000                0.125000        9.000000

Statistics summary for shard 3
==================================
Total data points: 10497
              size           gas  transaction_per_second  time_per_block
mean    745.131276     10.002858                0.000057        8.229875
std     703.301050    458.235708                0.002611        0.420772
min     738.000000      0.000000                0.000000        8.000000
25%     738.000000      0.000000                0.000000        8.000000
50%     738.000000      0.000000                0.000000        8.000000
75%     738.000000      0.000000                0.000000        8.000000
max   72777.000000  21000.000000                0.125000        9.000000

Features vs Time
PNG saved in ../../graphs/test_logs/mainnet/2020_04_25/
==================================

HTML saved in 

https://harmony-one.github.io/harmony-log-analysis/graphs/test_logs/mainnet/2020_04_25/transaction_per_second_vs_time.html

HTML saved in 

https://harmony-one.github.io/harmony-log-analysis/graphs/test_logs/mainnet/2020_04_25/time_per_block_vs_time.html

HTML saved in 

https://harmony-one.github.io/harmony-log-analysis/graphs/test_logs/mainnet/2020_04_25/size_vs_time.html

HTML saved in 

https://harmony-one.github.io/harmony-log-analysis/graphs/test_logs/mainnet/2020_04_25/gas_vs_time.html