Questions tagged [python]
Use for data science questions related to the programming language Python. Not intended for general coding questions (which should be asked on Stack Overflow).
6,620 questions
0
votes
0
answers
5
views
EDA before or after cleaning the dataset? What's the tradeoff and best practice?
I've read somewhere (I forgot the source) that we perform 2 types of EDA:
Light EDA:
See shapes of data with df.shape
See null with df.isna().mean() or df.isna().sum
See duplicates with df.duplicated(...
- 1
0
votes
0
answers
11
views
CLV Estimation: BTYD vs. Survival Analysis
I'm working on a project using the Elo Merchant Category Recommendation dataset (Kaggle). My goal is to perform Customer Segmentation base on their transactions by combining RFM metrics with Customer ...
0
votes
0
answers
11
views
Neural network (NN) model loss is increasing with epochs
I am trying to make a Convolutional Neural Network (CNN) using only numpy and other basic libraries. As a first step though, I needed to test the regular NN layers first, so I currently have a regular ...
- 1
7
votes
1
answer
619
views
Handling Profanity Censorship in BERTopic
I'm currently working in a dataset with censorship on profanity. Basically, fuck would be 4 heart emojis. Considering I'm trying to run a topic modelling w/ BERTopic, what kinda of preprocessing would ...
1
vote
0
answers
20
views
What causes a model to have such an output?
I'm training CSDI model and the output is very suspicious?
Low diffusion steps? too high learning rate? nothing seems to change this behavior? some normalization issue that I'm not accounting for, I'...
- 151
5
votes
1
answer
204
views
How to efficiently merge multiple CSV and JSON files into a single DataFrame using Pandas in Python
I am working with multiple data files in a folder where some files are in CSV format and others are in JSON format. I want to combine all of them into a single DataFrame for further analysis.
Here is ...
- 51
2
votes
0
answers
15
views
Trying to understand if I'm implementing GluonTS sliding window splitting and validation set correctly
The documentation is a little bit confusing so I thought I would ask here to make sure, I'm using:
...
- 151
5
votes
1
answer
55
views
Model's forecasts are not anchored correctly to the history
I'm implementing this paper
and trying to train it on a generated data and return full ground truths and a single forecast but the forecasts my model is producing are not anchored to the past series ...
- 151
7
votes
4
answers
339
views
Calculating next row in binary matrix
if I have the binary matrix which looks something like this (this is only 10 rows of binary matrix, I have a dataset of a million rows, so you can see what the binary matrix looks like):
...
2
votes
0
answers
46
views
GAN training (RGB to IR) High-quality results on training set, but blurry/hallucinated outputs on unseen test images
I am a 6th sem student and my mini project is IR pedestrian detection using yolo v8. My job is to train GAN(Pix2Pix-based) for the generation of synthetic IR images. I will give my code below. My 99th ...
- 21
5
votes
0
answers
32
views
Unable to predict values for test data
I have build and trained a NMT model using Rnn in Google colab and Now when I am trying to predict for my test data my Google colab session keeps on crashing . The shape of my test data is 47838×55
...
- 51
6
votes
1
answer
131
views
When attempting to maximize F1 score for a decision tree on test data using cost-complexity pruning why is it yielding the fully grown tree?
I'm learning about classification using decision trees. I'm using DecisionTreeClassifier function in the scikit-learn library in Python to train the model on training data (yields fully grown tree), ...
- 63
2
votes
0
answers
33
views
graph analysis for 2-dimensional edges for Natural Language Processing
Given a text resource (Corpus/novel/...) I want to find pair of words that 1) appear statistically significantly together and 2) extract contextual knowledge from these pairs. For simplicity I'm ...
5
votes
1
answer
120
views
Unable to run pandas/modin[ray] code on sagemaker unified studio
I am working on a movie recommendation problem where I get multiple files from the source, and the total data size is around 900 MB. I am using the ...
- 259
5
votes
1
answer
82
views
Python Datascience on windows OS behind a proxy impossible to cross: which solution do you use?
In an structure, IT has deployed a strict proxy policy (no specific right for any people).
Windows 11 is the OS installed for every people, in a strict way.
To run datascience tasks using python, in ...
- 51
