Category : Python

How to reshape and subset data with pandas functions?

     Pandas, Python, Worknotes

A beautiful extreme wide aerial view of snowy mountains

It is easy to change the layout, sort, reindex, rename, and subset table data using pandas commands. The simple code below shows you how to do this easily. The functions include melt(),pivot,sort_values,rename,sort_index,drop,filter,query,iloc,loc,iat,at and drop_duplicates. Prepare data #dowonload https://github.com/ziwangdeng/Data/blob/main/Vancouver_weather2010to2019_v00.csv import pandas as pd df=pd.read_csv(‘Vancouver_weather2010to2019_v00.csv’) cols=df.columns df1=df[cols[:10]] df2=df[cols[10:]] ll=len(df) df3=df.head(5000) df4=df.tail(ll-5000) df.columns Index([‘Unnamed: 0’, ‘Longitude (x)’, ‘Latitude ..

Read more

How to explore your csv file with pandas?

     Pandas, Python, Uncategorized, Worknotes

Beautiful happy picturesque charming lone maple trees

A comprehensive understanding of the overall situation of the data is the first step in data analysis. The following sample code shows you how to do this simply with pandas. Use of functions columns, keys(),axes,dtypes,info(),describe(),describe(include=object),isna().sum(),nunique(),value_counts(),len,shape,nsmallest(), nlargest(),sample(),head and tail. Read data #dowonload https://github.com/ziwangdeng/Data/blob/main/Vancouver_weather2010to2019_v00.csv import pandas as pd df=pd.read_csv(‘Vancouver_weather2010to2019_v00.csv’) Check column names df.columns Index([‘Unnamed: 0’, ‘Longitude (x)’, ..

Read more

How to avoid overfitting in deep learning with regularizer and drop out?

     AI, Data, Deep Learning, Keras, Machine Learning, Numpy, Pandas, Python, Scikit-learn, Worknotes

beautiful sprawling fantasy vast landscape

Sample code for reducing overfitting problems in deep learning.Answer the following questions: import numpy as np import pandas as pd from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler # create range of monthly dates download_dates = pd.date_range(start=’2019-01-01′, end=’2020-01-01′, freq=’MS’) # URL from Chrome DevTools Console base_url = (“https://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&” “stationID=51442&Year={}&Month={}&Day=7&timeframe=1&submit=Download+Data”) # add format option to year ..

Read more

How to construct a regression model with Keras with more than two input layers?

     AI, Data, Keras, Numpy, Pandas, Python, Scikit-learn, Tensorflow, Worknotes

Example code for a regression model with multiple layers. In addition to the input of the first layer, it keeps adding new inputs to the later layers. Prepare data import numpy as np import pandas as pd from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler # create range of monthly dates download_dates = pd.date_range(start=’2019-01-01′, end=’2020-01-01′, ..

Read more

How to optimize hyperparameter of DNN with Keras Tuner and AutoKeras?

     AI, Deep Learning, Keras, Machine Learning, Python, Scikit-learn, Tensorflow, Worknotes

deep ocean scape

Simple example code on hyperparameter optimization for DNN regression models. Prepare data import numpy as np import pandas as pd from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler # create range of monthly dates download_dates = pd.date_range(start=’2019-01-01′, end=’2020-01-01′, freq=’MS’) # URL from Chrome DevTools Console base_url = (“https://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&” “stationID=51442&Year={}&Month={}&Day=7&timeframe=1&submit=Download+Data”) # add format option to year ..

Read more

How to develope a regression model with Keras?

     AI, Data, Deep Learning, Keras, Machine Learning, Pandas, Python, Scikit-learn, Tensorflow, Worknotes

trees in spring, summer, autumn and winter

Example code for developing a regression model with keras. It can also answer following questions: Prepare data Read data import numpy as np import pandas as pd import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split # create range of monthly dates download_dates = pd.date_range(start=’2019-01-01′, end=’2020-01-01′, freq=’MS’) # URL from Chrome DevTools Console base_url = (“https://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&” ..

Read more

How to apply Scikit-learn transformers to do feature engineering?

     AI, Data Analytics, Deep Learning, Machine Learning, Python, Scikit-learn, Worknotes

Beautiful trees landscape image by AI

Example code for transforming a selected group of variables with Sklearn Transformer Wrapper. It can also answer following questions: Prepare data sample import numpy as np import pandas as pd import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from sklearn.impute import SimpleImputer from sklearn.preprocessing import StandardScaler from sklearn.preprocessing import OneHotEncoder from sklearn.preprocessing import PolynomialFeatures from ..

Read more

How to select features from dataset?

     AI, Data, Data Analytics, Deep Learning, Machine Learning, Pandas, Python, Worknotes

Beautiful forest, pond and sunshine

Example code for identifying and selecting the high predictive performance features from dataset for machine learning and deep learning models. It can also answer the following questions: Prepare data Code import numpy as np import pandas as pd import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split # create range of monthly dates download_dates = pd.date_range(start=’2019-01-01′, ..

Read more

How to transform continuous numerical variables into discrete variables?

     AI, Data, Deep Learning, Machine Learning, Numpy, Pandas, Python, Scikit-learn, Worknotes

Example code to transform continuous numerical variables into discrete variables with different methods. It cab also answer the following questions. Prepare data and load functions Code import numpy as np import pandas as pd import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from feature_engine.discretisation import EqualFrequencyDiscretiser from feature_engine.discretisation import EqualWidthDiscretiser from feature_engine.discretisation import ArbitraryDiscretiser from ..

Read more

How to create and add new features to the dataframe with feature-engine?

     AI, Data, Deep Learning, Machine Learning, Numpy, Pandas, Python, Scikit-learn, Worknotes

A scenic autumn landscape

Example code for creating and adding new features to a data frame using the feature-engine. It also answer following questions: Math features Code import numpy as np import pandas as pd import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from feature_engine.creation import MathFeatures from feature_engine.creation import RelativeFeatures from feature_engine.creation import CyclicalFeatures # create range of ..

Read more