goals

python 3 days, 11 hours ago

—————————————————————————————————————————— Fixing Autodoc excel readability issue and moving to production usage. Created new implementation for excel sheet to open and load all formulas automatically and then run autodoc PPT generation. Discussed with stakeholders and made the scripts to work out the best way to move it production runs. Created packaging configurations so that it will be uploaded to nexus and anyone can use it by just install using pip install autodoc-pptx Onboarding snowflake datasets to Anamolo Understood the complete flow from snowflake to Anamolo and alerting of the anomalies/data checks. Migrated snowflake data tables to the anamolo and configured the tables to run checks on daily basis so that everything will be tracked over time. Worked on anamolo APIs to automate the table configuration process with python, worked on core apis and its usage to understand automation. Implemented unit tests to cover over 90% of the code coverage and fixed all sonar checks. Conduct KT and sessions for team and interns on Anamolo setup and configurations. I’ve conducted lot of KT sessions and presentations to the team to educate how anamolo will work and how to configure everything. I also gone through the lot of DQ checks which are written in python and documented how to configure the same in anamolo as well. —————————————————————————————————————————— Question What knowledge or skills did you develop this year that helped you achieve positive results? —————————————————————————————————————————— In the past year, I have focused on learning and automating processes related to Anomalo—a powerful data quality monitoring tool. Anomalo has become essential for detecting and diagnosing data quality issues in large datasets. To enhance the efficiency and reliability of my tasks, I combined this knowledge with Python automation. Here’s how this skill has contributed to positive results: Understanding Anomalo's Usage Anomalo excels at identifying data quality issues by analyzing anomalies, missing data, schema changes, and trends. By deeply learning its functionalities, I have been able to: * Set up automated anomaly detection to monitor data health continuously, without manual intervention. * Understand various data quality metrics and alerts Anomalo offers, allowing me to interpret results more accurately and act on them quickly. * Customize monitoring configurations based on specific business needs, which enables a proactive response to potential data issues before they impact downstream processes. Developing Python Automation for Anomalo I used Python to automate many of Anomalo's manual processes, enhancing productivity and reliability. This automation skillset has allowed me to: * Automate report generation: By scripting automated reports in Python, I saved hours of manual work and provided stakeholders with up-to-date data quality insights. * Integrate anomaly alerts into workflow systems (like Slack or email), ensuring quick responses to any issues. * Create scripts to handle repetitive tasks, such as re-running checks, setting up new monitoring rules, or updating alert parameters, which increased operational efficiency. * Analyze and visualize data trends from Anomalo’s output, helping to identify long-term patterns and root causes of recurrent data issues. —————————————————————————————————————————— An overall rating is required. When assigning an overall rating, consider the following: * Contributions to Risk, Compliance, and Issue Management * Results Delivered * Demonstration of the Discover Behaviors Note: The Overall Rating cannot be more than one rating higher than the Required Risk Goal rating. —————————————————————————————————————————— 1. Enhanced Autodoc Excel Functionality for Production: Resolved readability issues with Autodoc Excel, created a seamless process for formula loading, and transitioned the Autodoc PPT generation tool to production. Configured packaging for easy access through Nexus and simplified usage with pip installation. 2. Automated Snowflake Dataset Integration with Anamolo: Developed a robust workflow for onboarding Snowflake data into Anamolo, enabling continuous data monitoring. Automated table configuration processes using Anamolo’s APIs, ensuring daily checks and seamless anomaly tracking. 3. Improved Code Quality and Coverage: Worked on Anamolo’s core APIs to automate data processes with Python, achieving over 90% code coverage through comprehensive unit tests and resolving all SonarQube checks for quality assurance. 4. Knowledge Transfer and Training: Conducted multiple knowledge transfer sessions for team members and interns, covering Anamolo setup, configuration, and Python-based data quality checks. Documented the configuration process for standardized use across the team. 5. Stakeholder Collaboration: Engaged with stakeholders to refine automation scripts, ensuring optimal performance in production. Collaborated to align implementation with user needs, enhancing tool reliability and user satisfaction.

kishore_kumar

Posted By

Please Login to Comment Here


                            #!/usr/bin/python
  
import boto

python aws beanstalk

sandeep


                            import os

def list_files_folders(path):

python python-os

kishore_kumar


                            import os
env = os.environ

python python-os

kishore_kumar


                            import os
print os.uname()
# Don't use os.system('uname -a'), its j

python python-os

kishore_kumar


                            def file_stats(path):
    f = open(path, 'r')
    lines = f.readlines()

python

kishore_kumar


                            def get_double(num):
    return num * 2

python

kishore_kumar


                            print "Welcome to python"

python

gaya38


                            a=input("Enter a value:")
if (a%2==0):
    print "The given number is even numb

python

gaya38


                            a=input("Enter a value:")
k=0
b=(a/2)+1

python

gaya38


                            import sys
x=len(sys.argv)
a=[]

python

gaya38


                            a = [1,43,98,5]#Dummy data
for l in range(len(a)-1):
        if (a[l]>a[l+1]):

python

gaya38


                            n=100
a=[10,20,30,40,50]
b=[60,70,80,90]

python

gaya38


                            n=input("Enter the constraint to print n
m=input("Enter the maximum value to prin
a=0

python

gaya38


                            import random
print random.uniform(10,500)

python

gaya38


                            import random;
z = [1,90,4,2]
z = random.shuffle(z)

python

gaya38


                            import requests

req = requests.get("https://httpbin.org/

python python-requests

kishore_kumar


                            sample_dict = { "number": 1, "fruits": [

for key in sample_dict:

python

kishore_kumar


                            import numpy as np

matrix = [[1,2,3], [4,5,6], [7,8,9]]

python numpy

kishore_kumar


                            import numpy as np

random_arr = np.random.randint(1,50,9)

python numpy

kishore_kumar


                            import numpy as np

random_arr = np.random.randint(1,50,9)

python numpy

kishore_kumar

goals

0 Comments

Blog

SiteMap

Social

KoderPlace