SDPR

In this notebook, we will use SDPR to calculate the PRS (Polygenic Risk Score).

Note: SDPR needs to be installed or placed in the same directory as this notebook.

SDPR Installation

To install SDPR, use the following command:

git clone https://github.com/eldronzhou/SDPR.git

Then, copy all the files to the current working directory.

Reference Panels

From the SDPR documentation:

If you are working with summary statistics from European (EUR) ancestry, you can download the MHC-included reference LD directory, which consists of 1 million HapMap3 SNPs estimated from 503 1000 Genome EUR samples, from the following links:

• MHC Included

• MHC Removed

You can also create the reference LD for your preferred reference panel by running the command below:

create_directory("EUR_MHC")
# os.system("tar -xzvf 1000G_EUR_MHC.tar.gz -C " + "EUR_MHC")

create_directory("EUR_noMHC")
# os.system("tar -xzvf 1000G_EUR_noMHC.tar.gz -C " + "EUR_noMHC")

Summary Statistics Format

The -ss option (required for -make_ref) specifies the path to the summary statistics file. It is recommended to follow the pipeline to clean up summary statistics using munge_sumstats.py.

The summary statistics file must have the following format (you can change the names in the header line):

SNP         A1      A2      BETA    P
rs737657    A       G       -2.044  0.0409
rs7086391   T       C       -2.257  0.024
rs1983865   T       C       3.652   0.00026

Examples

Estimating LD

SDPR -make_ref -ref_prefix ./test/1kg -ref_dir ./test/ref -chr 1 -make_ref

Performing MCMC

SDPR -mcmc -ss ./test/ss/ss.txt -ref_dir ./test/ref -chr 1 -out ./test/SDPR_out.txt

Hyperparemters

SDPR -options

Parameter	Description	Default
-make-ref	Estimate reference LD matrix.	-
-mcmc	Perform MCMC.	-
-ss	Path to the summary statistics file (required for -make_ref).	-
-ref_prefix	Path to the prefix of the bim file for the reference panel (required for -make_ref).	-
-ref_dir	Path to the directory containing the reference LD information (required).	-
-valid	Path to the bim file for the testing dataset (optional).	-
-out	Path to the output file containing estimated effect sizes (required for -mcmc).	-
-chr	Chromosome to work on (required). Currently supports 1-22.	-
-N	GWAS sample size (required for -mcmc).	-
-M	Max number of variance components. Must be greater than 4.	1000
-opt_llk	Which likelihood to evaluate. 1 for vanilla modified likelihood and 2 for SNPs genotyped on different individuals.	1
-iter	Number of iterations for MCMC.	1000
-burn	Number of burn-in iterations for MCMC.	200
-thin	Thinning for MCMC.	1
-n_threads	Number of threads to use.	1
-r2	r2 cut-off for partition of independent blocks.	0.1
-a	Factor to shrink the reference LD matrix.	0.1
-c	Factor to correct for the deflation.	1
-a0k	Hyperparameter for inverse gamma distribution.	0.5
-b0k	Hyperparameter for inverse gamma distribution.	0.5
-h	Print the options.	-

GWAS file processing for SDPR .

When the effect size relates to disease risk and is thus given as an odds ratio (OR) rather than BETA (for continuous traits), the PRS is computed as a product of ORs. To simplify this calculation, take the natural logarithm of the OR so that the PRS can be computed using summation instead.

For continuous phenotype GWAS, the SampleData1/SampleData1.gz file should have BETAs, and for binary phenotypes, it should have OR instead of BETAs. If BETAs are not available, we convert OR to BETAs using BETA = np.log(OR) and convert BETAs to OR using OR = np.exp(BETA).

import numpy as np; np is the NumPy module.

import os
import pandas as pd
import numpy as np

#filedirec = sys.argv[1]

filedirec = "SampleData1"
#filedirec = "asthma_19"
#filedirec = "migraine_0"

def check_phenotype_is_binary_or_continous(filedirec):
    # Read the processed quality controlled file for a phenotype
    df = pd.read_csv(filedirec+os.sep+filedirec+'_QC.fam',sep="\s+",header=None)
    column_values = df[5].unique()
 
    if len(set(column_values)) == 2:
        return "Binary"
    else:
        return "Continous"



# Read the GWAS file.
GWAS = filedirec + os.sep + filedirec+".gz"
df = pd.read_csv(GWAS,compression= "gzip",sep="\s+")
print(df.head().to_markdown()) 


if "BETA" in df.columns.to_list():
    # For Continous Phenotype.
    df = df[['CHR', 'BP', 'SNP', 'A1', 'A2', 'N', 'SE', 'P', 'BETA', 'INFO', 'MAF']]
else:
    df["BETA"] = np.log(df["OR"])
    df["SE"] = df["SE"]/df["OR"]
    df = df[['CHR', 'BP', 'SNP', 'A1', 'A2', 'N', 'SE', 'P', 'BETA', 'INFO', 'MAF']]
    
print(df.head().to_markdown()) 


 
df_transformed = pd.DataFrame({
    'SNP': df['SNP'],
    'A1': df['A1'],
    'A2': df['A2'],
    'BETA': df['BETA'],
    'P': df['P'],
})

n_gwas = df["N"].mean()    
df_transformed.to_csv(filedirec + os.sep +filedirec+".SDPR",sep="\t",index=False)
 
print(df_transformed.head().to_markdown())
print("Length of DataFrame!",len(df))

|    |   CHR |      BP | SNP        | A1   | A2   |      N |         SE |          P |         BETA |     INFO |      MAF |
|---:|------:|--------:|:-----------|:-----|:-----|-------:|-----------:|-----------:|-------------:|---------:|---------:|
|  0 |     1 |  849998 | rs13303222 | A    | G    | 388028 | 0.00264649 | 0.221234   |  0.00323734  | 0.913359 | 0.435377 |
|  1 |     1 |  881627 | rs2272757  | G    | A    | 388028 | 0.00213819 | 0.00109761 | -0.0069796   | 0.886698 | 0.398851 |
|  2 |     1 | 1002434 | rs11260596 | T    | C    | 388028 | 0.00203814 | 0.73213    |  0.000697641 | 0.90778  | 0.38935  |
|  3 |     1 | 1062638 | rs9442373  | C    | A    | 388028 | 0.00204733 | 0.30861    |  0.00208448  | 0.910038 | 0.498987 |
|  4 |     1 | 1094979 | rs7538773  | G    | A    | 388028 | 0.00209216 | 0.320391   |  0.00207889  | 0.884386 | 0.441638 |
|    |   CHR |      BP | SNP        | A1   | A2   |      N |         SE |          P |         BETA |     INFO |      MAF |
|---:|------:|--------:|:-----------|:-----|:-----|-------:|-----------:|-----------:|-------------:|---------:|---------:|
|  0 |     1 |  849998 | rs13303222 | A    | G    | 388028 | 0.00264649 | 0.221234   |  0.00323734  | 0.913359 | 0.435377 |
|  1 |     1 |  881627 | rs2272757  | G    | A    | 388028 | 0.00213819 | 0.00109761 | -0.0069796   | 0.886698 | 0.398851 |
|  2 |     1 | 1002434 | rs11260596 | T    | C    | 388028 | 0.00203814 | 0.73213    |  0.000697641 | 0.90778  | 0.38935  |
|  3 |     1 | 1062638 | rs9442373  | C    | A    | 388028 | 0.00204733 | 0.30861    |  0.00208448  | 0.910038 | 0.498987 |
|  4 |     1 | 1094979 | rs7538773  | G    | A    | 388028 | 0.00209216 | 0.320391   |  0.00207889  | 0.884386 | 0.441638 |
|    | SNP        | A1   | A2   |         BETA |          P |
|---:|:-----------|:-----|:-----|-------------:|-----------:|
|  0 | rs13303222 | A    | G    |  0.00323734  | 0.221234   |
|  1 | rs2272757  | G    | A    | -0.0069796   | 0.00109761 |
|  2 | rs11260596 | T    | C    |  0.000697641 | 0.73213    |
|  3 | rs9442373  | C    | A    |  0.00208448  | 0.30861    |
|  4 | rs7538773  | G    | A    |  0.00207889  | 0.320391   |
Length of DataFrame! 36567

Define Hyperparameters

Define hyperparameters to be optimized and set initial values.

Extract Valid SNPs from Clumped File

For Windows, download gwak, and for Linux, the awk command is sufficient. For Windows, GWAK is required. You can download it from here. Get it and place it in the same directory.

Execution Path

At this stage, we have the genotype training data newtrainfilename = "train_data.QC" and genotype test data newtestfilename = "test_data.QC".

We modified the following variables:

1. filedirec = "SampleData1" or filedirec = sys.argv[1]

2. foldnumber = "0" or foldnumber = sys.argv[2] for HPC.

Only these two variables can be modified to execute the code for specific data and specific folds. Though the code can be executed separately for each fold on HPC and separately for each dataset, it is recommended to execute it for multiple diseases and one fold at a time. Here’s the corrected text in Markdown format:

P-values

PRS calculation relies on P-values. SNPs with low P-values, indicating a high degree of association with a specific trait, are considered for calculation.

You can modify the code below to consider a specific set of P-values and save the file in the same format.

We considered the following parameters:

• Minimum P-value: 1e-10

• Maximum P-value: 1.0

• Minimum exponent: 10 (Minimum P-value in exponent)

• Number of intervals: 100 (Number of intervals to be considered)

The code generates an array of logarithmically spaced P-values:

import numpy as np
import os

minimumpvalue = 10  # Minimum exponent for P-values
numberofintervals = 100  # Number of intervals to be considered

allpvalues = np.logspace(-minimumpvalue, 0, numberofintervals, endpoint=True)  # Generating an array of logarithmically spaced P-values

print("Minimum P-value:", allpvalues[0])
print("Maximum P-value:", allpvalues[-1])

count = 1
with open(os.path.join(folddirec, 'range_list'), 'w') as file:
    for value in allpvalues:
        file.write(f'pv_{value} 0 {value}\n')  # Writing range information to the 'range_list' file
        count += 1

pvaluefile = os.path.join(folddirec, 'range_list')

In this code:

• minimumpvalue defines the minimum exponent for P-values.

• numberofintervals specifies how many intervals to consider.

• allpvalues generates an array of P-values spaced logarithmically.

• The script writes these P-values to a file named range_list in the specified directory.

from operator import index
import pandas as pd
import numpy as np
import os
import subprocess
import sys
import pandas as pd
import statsmodels.api as sm
import pandas as pd
from sklearn.metrics import roc_auc_score, confusion_matrix
from statsmodels.stats.contingency_tables import mcnemar

def create_directory(directory):
    """Function to create a directory if it doesn't exist."""
    if not os.path.exists(directory):  # Checking if the directory doesn't exist
        os.makedirs(directory)  # Creating the directory if it doesn't exist
    return directory  # Returning the created or existing directory

 
#foldnumber = sys.argv[1]
foldnumber = "0"  # Setting 'foldnumber' to "0"

folddirec = filedirec + os.sep + "Fold_" + foldnumber  # Creating a directory path for the specific fold
trainfilename = "train_data"  # Setting the name of the training data file
newtrainfilename = "train_data.QC"  # Setting the name of the new training data file

testfilename = "test_data"  # Setting the name of the test data file
newtestfilename = "test_data.QC"  # Setting the name of the new test data file

# Number of PCA to be included as a covariate.
numberofpca = ["6"]  # Setting the number of PCA components to be included

# Clumping parameters.
clump_p1 = [1]  # List containing clump parameter 'p1'
clump_r2 = [0.1]  # List containing clump parameter 'r2'
clump_kb = [200]  # List containing clump parameter 'kb'

# Pruning parameters.
p_window_size = [200]  # List containing pruning parameter 'window_size'
p_slide_size = [50]  # List containing pruning parameter 'slide_size'
p_LD_threshold = [0.25]  # List containing pruning parameter 'LD_threshold'

# Kindly note that the number of p-values to be considered varies, and the actual p-value depends on the dataset as well.
# We will specify the range list here.

minimumpvalue = 10  # Minimum p-value in exponent
numberofintervals = 20  # Number of intervals to be considered
allpvalues = np.logspace(-minimumpvalue, 0, numberofintervals, endpoint=True)  # Generating an array of logarithmically spaced p-values



count = 1
with open(folddirec + os.sep + 'range_list', 'w') as file:
    for value in allpvalues:
        file.write(f'pv_{value} 0 {value}\n')  # Writing range information to the 'range_list' file
        count = count + 1

pvaluefile = folddirec + os.sep + 'range_list'

 

Define Helper Functions

1. Perform Clumping and Pruning

2. Calculate PCA Using Plink

3. Fit Binary Phenotype and Save Results

4. Fit Continuous Phenotype and Save Results

import os
import subprocess
import pandas as pd
import statsmodels.api as sm
from sklearn.metrics import explained_variance_score


def perform_clumping_and_pruning_on_individual_data(traindirec, newtrainfilename,numberofpca, p1_val, p2_val, p3_val, c1_val, c2_val, c3_val,Name,pvaluefile):
    
    command = [
    "./plink",
    "--bfile", traindirec+os.sep+newtrainfilename,
    "--indep-pairwise", p1_val, p2_val, p3_val,
    "--out", traindirec+os.sep+trainfilename
    ]
    subprocess.run(command)
    # First perform pruning and then clumping and the pruning.

    command = [
    "./plink",
    "--bfile", traindirec+os.sep+newtrainfilename,
    "--clump-p1", c1_val,
    "--extract", traindirec+os.sep+trainfilename+".prune.in",
    "--clump-r2", c2_val,
    "--clump-kb", c3_val,
    "--clump", filedirec+os.sep+filedirec+".txt",
    "--clump-snp-field", "SNP",
    "--clump-field", "P",
    "--out", traindirec+os.sep+trainfilename
    ]    
    subprocess.run(command)

    # Extract the valid SNPs from th clumped file.
    # For windows download gwak for linux awk commmand is sufficient.
    ### For windows require GWAK.
    ### https://sourceforge.net/projects/gnuwin32/
    ##3 Get it and place it in the same direc.
    #os.system("gawk "+"\""+"NR!=1{print $3}"+"\"  "+ traindirec+os.sep+trainfilename+".clumped >  "+traindirec+os.sep+trainfilename+".valid.snp")
    #print("gawk "+"\""+"NR!=1{print $3}"+"\"  "+ traindirec+os.sep+trainfilename+".clumped >  "+traindirec+os.sep+trainfilename+".valid.snp")

    #Linux:
    command = f"awk 'NR!=1{{print $3}}' {traindirec}{os.sep}{trainfilename}.clumped > {traindirec}{os.sep}{trainfilename}.valid.snp"
    os.system(command)
    
    
    command = [
    "./plink",
    "--make-bed",
    "--bfile", traindirec+os.sep+newtrainfilename,
    "--indep-pairwise", p1_val, p2_val, p3_val,
    "--extract", traindirec+os.sep+trainfilename+".valid.snp",
    "--out", traindirec+os.sep+newtrainfilename+".clumped.pruned"
    ]
    subprocess.run(command)
    
    command = [
    "./plink",
    "--make-bed",
    "--bfile", traindirec+os.sep+testfilename,
    "--indep-pairwise", p1_val, p2_val, p3_val,
    "--extract", traindirec+os.sep+trainfilename+".valid.snp",
    "--out", traindirec+os.sep+testfilename+".clumped.pruned"
    ]
    subprocess.run(command)    
    
    
 
def calculate_pca_for_traindata_testdata_for_clumped_pruned_snps(traindirec, newtrainfilename,p):
    
    # Calculate the PRS for the test data using the same set of SNPs and also calculate the PCA.


    # Also extract the PCA at this point.
    # PCA are calculated afer clumping and pruining.
    command = [
        "./plink",
        "--bfile", folddirec+os.sep+testfilename+".clumped.pruned",
        # Select the final variants after clumping and pruning.
        "--extract", traindirec+os.sep+trainfilename+".valid.snp",
        "--pca", p,
        "--out", folddirec+os.sep+testfilename
    ]
    subprocess.run(command)


    command = [
    "./plink",
        "--bfile", traindirec+os.sep+newtrainfilename+".clumped.pruned",
        # Select the final variants after clumping and pruning.        
        "--extract", traindirec+os.sep+trainfilename+".valid.snp",
        "--pca", p,
        "--out", traindirec+os.sep+trainfilename
    ]
    subprocess.run(command)

# This function fit the binary model on the PRS.
def fit_binary_phenotype_on_PRS(traindirec, newtrainfilename, p, 
                        M_val,opt_llk_val,iter_val_val,burn_val,thin_val,r2_val,
                        a_val,c_val,a0k_val,b0k_val,referenceset, p1_val, p2_val, p3_val, c1_val, c2_val, c3_val,Name,pvaluefile):
    threshold_values = allpvalues

    # Merge the covariates, pca and phenotypes.
    tempphenotype_train = pd.read_table(traindirec+os.sep+newtrainfilename+".clumped.pruned"+".fam", sep="\s+",header=None)
    phenotype_train = pd.DataFrame()
    phenotype_train["Phenotype"] = tempphenotype_train[5].values
    pcs_train = pd.read_table(traindirec+os.sep+trainfilename+".eigenvec", sep="\s+",header=None, names=["FID", "IID"] + [f"PC{str(i)}" for i in range(1, int(p)+1)])
    covariate_train = pd.read_table(traindirec+os.sep+trainfilename+".cov",sep="\s+")
    covariate_train.fillna(0, inplace=True)
    covariate_train = covariate_train[covariate_train["FID"].isin(pcs_train["FID"].values) & covariate_train["IID"].isin(pcs_train["IID"].values)]
    covariate_train['FID'] = covariate_train['FID'].astype(str)
    pcs_train['FID'] = pcs_train['FID'].astype(str)
    covariate_train['IID'] = covariate_train['IID'].astype(str)
    pcs_train['IID'] = pcs_train['IID'].astype(str)
    covandpcs_train = pd.merge(covariate_train, pcs_train, on=["FID","IID"])
    covandpcs_train.fillna(0, inplace=True)


    ## Scale the covariates!
    from sklearn.preprocessing import MinMaxScaler
    from sklearn.metrics import explained_variance_score
    scaler = MinMaxScaler()
    normalized_values_train = scaler.fit_transform(covandpcs_train.iloc[:, 2:])
    #covandpcs_train.iloc[:, 2:] = normalized_values_test 
    
    
    tempphenotype_test = pd.read_table(traindirec+os.sep+testfilename+".clumped.pruned"+".fam", sep="\s+",header=None)
    phenotype_test= pd.DataFrame()
    phenotype_test["Phenotype"] = tempphenotype_test[5].values
    pcs_test = pd.read_table(traindirec+os.sep+testfilename+".eigenvec", sep="\s+",header=None, names=["FID", "IID"] + [f"PC{str(i)}" for i in range(1, int(p)+1)])
    covariate_test = pd.read_table(traindirec+os.sep+testfilename+".cov",sep="\s+")
    covariate_test.fillna(0, inplace=True)
    covariate_test = covariate_test[covariate_test["FID"].isin(pcs_test["FID"].values) & covariate_test["IID"].isin(pcs_test["IID"].values)]
    covariate_test['FID'] = covariate_test['FID'].astype(str)
    pcs_test['FID'] = pcs_test['FID'].astype(str)
    covariate_test['IID'] = covariate_test['IID'].astype(str)
    pcs_test['IID'] = pcs_test['IID'].astype(str)
    covandpcs_test = pd.merge(covariate_test, pcs_test, on=["FID","IID"])
    covandpcs_test.fillna(0, inplace=True)
    normalized_values_test  = scaler.transform(covandpcs_test.iloc[:, 2:])
    #covandpcs_test.iloc[:, 2:] = normalized_values_test     
    
    
    
    
    tempalphas = [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9]
    l1weights = [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9]

    tempalphas = [0.1]
    l1weights = [0.1]

    phenotype_train["Phenotype"] = phenotype_train["Phenotype"].replace({1: 0, 2: 1}) 
    phenotype_test["Phenotype"] = phenotype_test["Phenotype"].replace({1: 0, 2: 1})
      
    for tempalpha in tempalphas:
        for l1weight in l1weights:

            
            try:
                null_model =  sm.Logit(phenotype_train["Phenotype"], sm.add_constant(covandpcs_train.iloc[:, 2:])).fit_regularized(alpha=tempalpha, L1_wt=l1weight)
                #null_model =  sm.Logit(phenotype_train["Phenotype"], sm.add_constant(covandpcs_train.iloc[:, 2:])).fit()
            
            except:
                print("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")
                continue

            train_null_predicted = null_model.predict(sm.add_constant(covandpcs_train.iloc[:, 2:]))
            
            from sklearn.metrics import roc_auc_score, confusion_matrix
            from sklearn.metrics import r2_score
            
            test_null_predicted = null_model.predict(sm.add_constant(covandpcs_test.iloc[:, 2:]))
            
           
            
            global prs_result 
            for i in threshold_values:
                try:
                    prs_train = pd.read_table(traindirec+os.sep+Name+os.sep+"train_data.pv_"+f"{i}.profile", sep="\s+", usecols=["FID", "IID", "SCORE"])
                except:
                    continue

                prs_train['FID'] = prs_train['FID'].astype(str)
                prs_train['IID'] = prs_train['IID'].astype(str)
                try:
                    prs_test = pd.read_table(traindirec+os.sep+Name+os.sep+"test_data.pv_"+f"{i}.profile", sep="\s+", usecols=["FID", "IID", "SCORE"])
                except:
                    continue
                prs_test['FID'] = prs_test['FID'].astype(str)
                prs_test['IID'] = prs_test['IID'].astype(str)
                pheno_prs_train = pd.merge(covandpcs_train, prs_train, on=["FID", "IID"])
                pheno_prs_test = pd.merge(covandpcs_test, prs_test, on=["FID", "IID"])
        
                try:
                    model = sm.Logit(phenotype_train["Phenotype"], sm.add_constant(pheno_prs_train.iloc[:, 2:])).fit_regularized(alpha=tempalpha, L1_wt=l1weight)
                    #model = sm.Logit(phenotype_train["Phenotype"], sm.add_constant(pheno_prs_train.iloc[:, 2:])).fit()
                
                except:
                    continue


                
                train_best_predicted = model.predict(sm.add_constant(pheno_prs_train.iloc[:, 2:]))    
 

                test_best_predicted = model.predict(sm.add_constant(pheno_prs_test.iloc[:, 2:])) 
 
        
                from sklearn.metrics import roc_auc_score, confusion_matrix

                prs_result = prs_result._append({
                    "clump_p1": c1_val,
                    "clump_r2": c2_val,
                    "clump_kb": c3_val,
                    "p_window_size": p1_val,
                    "p_slide_size": p2_val,
                    "p_LD_threshold": p3_val,
                    "pvalue": i,
                    "numberofpca":p, 

                    "tempalpha":str(tempalpha),
                    "l1weight":str(l1weight),
                    
                    "SDPR_M_val": M_val,
                    "SDPR_opt_llk_val": opt_llk_val,
                    "SDPR_iter_val_val": iter_val_val,
                    "SDPR_burn_val": burn_val,
                    "SDPR_thin_val": thin_val,
                    "SDPR_r2_val": r2_val,
                    "SDPR_a_val": a_val,
                    "SDPR_c_val": c_val,
                    "SDPR_a0k_val": a0k_val,
                    "SDPR_b0k_val": b0k_val,
                    "SDPR_referenceset": referenceset,

                    
                    

                    "Train_pure_prs":roc_auc_score(phenotype_train["Phenotype"].values,prs_train['SCORE'].values),
                    "Train_null_model":roc_auc_score(phenotype_train["Phenotype"].values,train_null_predicted.values),
                    "Train_best_model":roc_auc_score(phenotype_train["Phenotype"].values,train_best_predicted.values),
                    
                    "Test_pure_prs":roc_auc_score(phenotype_test["Phenotype"].values,prs_test['SCORE'].values),
                    "Test_null_model":roc_auc_score(phenotype_test["Phenotype"].values,test_null_predicted.values),
                    "Test_best_model":roc_auc_score(phenotype_test["Phenotype"].values,test_best_predicted.values),
                    
                }, ignore_index=True)

          
                prs_result.to_csv(traindirec+os.sep+Name+os.sep+"Results.csv",index=False)
     
    return

# This function fit the binary model on the PRS.
def fit_continous_phenotype_on_PRS(traindirec, newtrainfilename, p, 
                        M_val,opt_llk_val,iter_val_val,burn_val,thin_val,r2_val,
                        a_val,c_val,a0k_val,b0k_val,referenceset,p1_val, p2_val, p3_val, c1_val, c2_val, c3_val,Name,pvaluefile):
    threshold_values = allpvalues

    # Merge the covariates, pca and phenotypes.
    tempphenotype_train = pd.read_table(traindirec+os.sep+newtrainfilename+".clumped.pruned"+".fam", sep="\s+",header=None)
    phenotype_train = pd.DataFrame()
    phenotype_train["Phenotype"] = tempphenotype_train[5].values
    pcs_train = pd.read_table(traindirec+os.sep+trainfilename+".eigenvec", sep="\s+",header=None, names=["FID", "IID"] + [f"PC{str(i)}" for i in range(1, int(p)+1)])
    covariate_train = pd.read_table(traindirec+os.sep+trainfilename+".cov",sep="\s+")
    covariate_train.fillna(0, inplace=True)
    covariate_train = covariate_train[covariate_train["FID"].isin(pcs_train["FID"].values) & covariate_train["IID"].isin(pcs_train["IID"].values)]
    covariate_train['FID'] = covariate_train['FID'].astype(str)
    pcs_train['FID'] = pcs_train['FID'].astype(str)
    covariate_train['IID'] = covariate_train['IID'].astype(str)
    pcs_train['IID'] = pcs_train['IID'].astype(str)
    covandpcs_train = pd.merge(covariate_train, pcs_train, on=["FID","IID"])
    covandpcs_train.fillna(0, inplace=True)


    ## Scale the covariates!
    from sklearn.preprocessing import MinMaxScaler
    from sklearn.metrics import explained_variance_score
    scaler = MinMaxScaler()
    normalized_values_train = scaler.fit_transform(covandpcs_train.iloc[:, 2:])
    #covandpcs_train.iloc[:, 2:] = normalized_values_test 
    
    tempphenotype_test = pd.read_table(traindirec+os.sep+testfilename+".clumped.pruned"+".fam", sep="\s+",header=None)
    phenotype_test= pd.DataFrame()
    phenotype_test["Phenotype"] = tempphenotype_test[5].values
    pcs_test = pd.read_table(traindirec+os.sep+testfilename+".eigenvec", sep="\s+",header=None, names=["FID", "IID"] + [f"PC{str(i)}" for i in range(1, int(p)+1)])
    covariate_test = pd.read_table(traindirec+os.sep+testfilename+".cov",sep="\s+")
    covariate_test.fillna(0, inplace=True)
    covariate_test = covariate_test[covariate_test["FID"].isin(pcs_test["FID"].values) & covariate_test["IID"].isin(pcs_test["IID"].values)]
    covariate_test['FID'] = covariate_test['FID'].astype(str)
    pcs_test['FID'] = pcs_test['FID'].astype(str)
    covariate_test['IID'] = covariate_test['IID'].astype(str)
    pcs_test['IID'] = pcs_test['IID'].astype(str)
    covandpcs_test = pd.merge(covariate_test, pcs_test, on=["FID","IID"])
    covandpcs_test.fillna(0, inplace=True)
    normalized_values_test  = scaler.transform(covandpcs_test.iloc[:, 2:])
    #covandpcs_test.iloc[:, 2:] = normalized_values_test     
    
    
    
    
    tempalphas = [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9]
    l1weights = [0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9]

    tempalphas = [0.1]
    l1weights = [0.1]

    #phenotype_train["Phenotype"] = phenotype_train["Phenotype"].replace({1: 0, 2: 1}) 
    #phenotype_test["Phenotype"] = phenotype_test["Phenotype"].replace({1: 0, 2: 1})
      
    for tempalpha in tempalphas:
        for l1weight in l1weights:

            
            try:
                #null_model =  sm.OLS(phenotype_train["Phenotype"], sm.add_constant(covandpcs_train.iloc[:, 2:])).fit_regularized(alpha=tempalpha, L1_wt=l1weight)
                null_model =  sm.OLS(phenotype_train["Phenotype"], sm.add_constant(covandpcs_train.iloc[:, 2:])).fit()
                #null_model =  sm.OLS(phenotype_train["Phenotype"], sm.add_constant(covandpcs_train.iloc[:, 2:])).fit()
            except:
                print("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")
                continue

            train_null_predicted = null_model.predict(sm.add_constant(covandpcs_train.iloc[:, 2:]))
            
            from sklearn.metrics import roc_auc_score, confusion_matrix
            from sklearn.metrics import r2_score
            
            test_null_predicted = null_model.predict(sm.add_constant(covandpcs_test.iloc[:, 2:]))
            
            
            
            global prs_result 
            for i in threshold_values:
                try:
                    prs_train = pd.read_table(traindirec+os.sep+Name+os.sep+"train_data.pv_"+f"{i}.profile", sep="\s+", usecols=["FID", "IID", "SCORE"])
                except:
                    continue

                prs_train['FID'] = prs_train['FID'].astype(str)
                prs_train['IID'] = prs_train['IID'].astype(str)
                try:
                    prs_test = pd.read_table(traindirec+os.sep+Name+os.sep+"test_data.pv_"+f"{i}.profile", sep="\s+", usecols=["FID", "IID", "SCORE"])
                except:
                    continue
                prs_test['FID'] = prs_test['FID'].astype(str)
                prs_test['IID'] = prs_test['IID'].astype(str)
                pheno_prs_train = pd.merge(covandpcs_train, prs_train, on=["FID", "IID"])
                pheno_prs_test = pd.merge(covandpcs_test, prs_test, on=["FID", "IID"])
        
                try:
                    #model = sm.OLS(phenotype_train["Phenotype"], sm.add_constant(pheno_prs_train.iloc[:, 2:])).fit_regularized(alpha=tempalpha, L1_wt=l1weight)
                    model = sm.OLS(phenotype_train["Phenotype"], sm.add_constant(pheno_prs_train.iloc[:, 2:])).fit()
                
                except:
                    continue


                
                train_best_predicted = model.predict(sm.add_constant(pheno_prs_train.iloc[:, 2:]))    
                test_best_predicted = model.predict(sm.add_constant(pheno_prs_test.iloc[:, 2:])) 
 
        
                from sklearn.metrics import roc_auc_score, confusion_matrix
             
                prs_result = prs_result._append({
                    "clump_p1": c1_val,
                    "clump_r2": c2_val,
                    "clump_kb": c3_val,
                    "p_window_size": p1_val,
                    "p_slide_size": p2_val,
                    "p_LD_threshold": p3_val,
                    "pvalue": i,
                    "numberofpca":p, 

                    
                    
                    "SDPR_M_val": M_val,
                    "SDPR_opt_llk_val": opt_llk_val,
                    "SDPR_iter_val_val": iter_val_val,
                    "SDPR_burn_val": burn_val,
                    "SDPR_thin_val": thin_val,
                    "SDPR_r2_val": r2_val,
                    "SDPR_a_val": a_val,
                    "SDPR_c_val": c_val,
                    "SDPR_a0k_val": a0k_val,
                    "SDPR_b0k_val": b0k_val,
                    "SDPR_referenceset": referenceset,


                    "tempalpha":str(tempalpha),
                    "l1weight":str(l1weight),
                     

                    "Train_pure_prs":explained_variance_score(phenotype_train["Phenotype"],prs_train['SCORE'].values),
                    "Train_null_model":explained_variance_score(phenotype_train["Phenotype"],train_null_predicted),
                    "Train_best_model":explained_variance_score(phenotype_train["Phenotype"],train_best_predicted),
                    
                    "Test_pure_prs":explained_variance_score(phenotype_test["Phenotype"],prs_test['SCORE'].values),
                    "Test_null_model":explained_variance_score(phenotype_test["Phenotype"],test_null_predicted),
                    "Test_best_model":explained_variance_score(phenotype_test["Phenotype"],test_best_predicted),
                    
                }, ignore_index=True)

          
                prs_result.to_csv(traindirec+os.sep+Name+os.sep+"Results.csv",index=False)
     
    return

Execute SDPR

# Define a global variable to store results
prs_result = pd.DataFrame()
def transform_sdpr_data(traindirec, newtrainfilename,numberofpca, M_val,opt_llk_val,iter_val_val,burn_val,thin_val,r2_val,a_val,c_val,a0k_val,b0k_val,referenceset,p1_val, p2_val, p3_val, c1_val, c2_val, c3_val,Name,pvaluefile):

    create_directory(traindirec+os.sep+"ref")
    create_directory(traindirec+os.sep+"result")
    
    # Delete files from the previous iteration.
    folder_path = os.path.join(traindirec, "result")

    # Create the directory if it does not exist
    if not os.path.exists(folder_path):
        os.makedirs(folder_path)

    import glob
    file_list = glob.glob(traindirec+os.sep+"result"+os.sep+"*")

    for f in file_list:
        if os.path.exists(f):
            os.remove(f)
            print(f"The file {f} removed")
        else:
            print(f"The file {f} does not exist")  
        
  
    
    file_path = os.path.join(traindirec, "train_data.SDPRNew")

    try:
        if os.path.exists(file_path):
            os.remove(file_path)
        else:
            print(f"The file {file_path} does not exist.")
    except Exception as e:
        print(f"An error occurred: {e}")        
    
    ### First perform clumping on the file and save the clumpled file.
    perform_clumping_and_pruning_on_individual_data(traindirec, newtrainfilename,p, p1_val, p2_val, p3_val, c1_val, c2_val, c3_val,Name,pvaluefile)
    
    #newtrainfilename = newtrainfilename+".clumped.pruned"
    #testfilename = testfilename+".clumped.pruned"
    
    
    #clupmedfile = traindirec+os.sep+newtrainfilename+".clump"
    #prunedfile = traindirec+os.sep+newtrainfilename+".clumped.pruned"

        
    # Also extract the PCA at this point for both test and training data.
    calculate_pca_for_traindata_testdata_for_clumped_pruned_snps(traindirec, newtrainfilename,p)

    #Extract p-values from the GWAS file.
    # Command for Linux.
    #os.system("awk "+"\'"+"{print $3,$8}"+"\'"+" ./"+filedirec+os.sep+filedirec+".txt >  ./"+traindirec+os.sep+"SNP.pvalue")

    # Command for windows.
    ### For windows get GWAK.
    ### https://sourceforge.net/projects/gnuwin32/
    ##3 Get it and place it in the same direc.
    #os.system("gawk "+"\""+"{print $3,$8}"+"\""+" ./"+filedirec+os.sep+filedirec+".txt >  ./"+traindirec+os.sep+"SNP.pvalue")
    #print("gawk "+"\""+"{print $3,$8}"+"\""+" ./"+filedirec+os.sep+filedirec+".txt >  ./"+traindirec+os.sep+"SNP.pvalue")

    #exit(0)
 

    for chromosome in range(1,23):
        plink_command = [
            "./plink",
            "--bfile", traindirec+os.sep+newtrainfilename+".clumped.pruned",
            "--chr", str(chromosome),
            "--make-bed",
            "--out", traindirec+os.sep+newtrainfilename+".clumped.pruned."+str(chromosome)
        ]
        subprocess.run(plink_command)
        if referenceset=="custom":
            create_directory(traindirec+os.sep+"ref")
            create_directory(traindirec+os.sep+"result")
            command = [
                "./SDPR",
                "-make_ref",
                "-ref_prefix", traindirec+os.sep+newtrainfilename+".clumped.pruned",
                "-chr", str(chromosome),
                "-ref_dir", traindirec+os.sep+"ref"
            ]
            subprocess.run(command)
            command = [
                "./SDPR",
                "-mcmc",
                "-n_threads", "3",
                "-ref_dir", traindirec+os.sep+"ref",
                "-ss", filedirec+os.sep+filedirec+".SDPR",
                "-N", str(int(n_gwas)),
                "-M",str(M_val),
                "-opt_llk",str(opt_llk_val),
                "-iter",str(iter_val_val),
                "-burn",str(burn_val),
                "-thin",str(thin_val),
                
                "-r2",str(r2_val),
                "-a",str(a_val),
                "-c",str(c_val),
                "-a0k",str(a0k_val),
                "-b0k",str(b0k_val),
                "-chr", str(chromosome),
                "-out", traindirec+os.sep+"result"+os.sep+str(chromosome)
                #"-out", "ref"+os.sep+str(chromosome)
            
            ]
            subprocess.run(command)
        else:
            command = [
                "./SDPR",
                "-mcmc",
                "-ref_dir", referenceset,
                "-ss", filedirec+os.sep+filedirec+".SDPR",
                "-N", str(int(n_gwas)),
                "-M",str(M_val),
                "-opt_llk",str(opt_llk_val),
                "-iter",str(iter_val_val),
                "-burn",str(burn_val),
                "-thin",str(thin_val),
                
                "-r2",str(r2_val),
                "-a",str(a_val),
                "-c",str(c_val),
                "-a0k",str(a0k_val),
                "-b0k",str(b0k_val),
                "-chr", str(chromosome),
                "-out", traindirec+os.sep+"result"+os.sep+str(chromosome)
            ]
            subprocess.run(command)            

            


    file_list = glob.glob(traindirec+os.sep+"result"+os.sep+"*")
    sorted_file_list = sorted(file_list, key=lambda x: int(''.join(filter(str.isdigit, x))))
    
    merged_df = pd.DataFrame()
    for file in sorted_file_list:
        print(file)
        df = pd.read_csv(file,sep="\s+")  # No header
        merged_df = pd.concat([merged_df, df], ignore_index=True)

    print(merged_df.head())
     
    
    num_columns = len(merged_df.columns)

    merged_df.columns = ["SNP","A1","NewBeta"]
 
    merged_df = merged_df[merged_df["NewBeta"]!="beta"]
    merged_df = merged_df.fillna(0)
 
    if check_phenotype_is_binary_or_continous(filedirec)=="Binary":
        merged_df["NewBeta"] = np.exp(merged_df["NewBeta"])
        pass
    
    else:
        pass    
    
    
    merged_df.to_csv(traindirec+os.sep+"train_data.SDPRNew",sep="\t",index=False)    
    
    # Caluclate Plink Score.
    command = [
        "./plink",
         "--bfile", traindirec+os.sep+newtrainfilename+".clumped.pruned",
        ### SNP column = 3, Effect allele column 1 = 4, OR column=9
        "--score", traindirec+os.sep+"train_data.SDPRNew", "1", "2", "3", "header",
        "--q-score-range", traindirec+os.sep+"range_list",traindirec+os.sep+"SNP.pvalue",
        "--extract", traindirec+os.sep+trainfilename+".valid.snp",
        "--out", traindirec+os.sep+Name+os.sep+trainfilename
    ]
    #exit(0)
    subprocess.run(command)
    


    command = [
        "./plink",
        "--bfile", folddirec+os.sep+testfilename+".clumped.pruned",
        ### SNP column = 3, Effect allele column 1 = 4, Beta column=12
        "--score", traindirec+os.sep+"train_data.SDPRNew", "1", "2", "3", "header",
        "--q-score-range", traindirec+os.sep+"range_list",traindirec+os.sep+"SNP.pvalue",
        "--extract", traindirec+os.sep+trainfilename+".valid.snp",
        "--out", folddirec+os.sep+Name+os.sep+testfilename
    ]
    subprocess.run(command)
     

    # At this stage the scores are finalizied. 
    # The next step is to fit the model and find the explained variance by each profile.

    # Load the PCA and Load the Covariates for trainingdatafirst.
    
    if check_phenotype_is_binary_or_continous(filedirec)=="Binary":
        print("Binary Phenotype!")
        fit_binary_phenotype_on_PRS(traindirec, newtrainfilename, p, 
                        M_val,opt_llk_val,iter_val_val,burn_val,thin_val,r2_val,
                        a_val,c_val,a0k_val,b0k_val,referenceset, p1_val, p2_val, p3_val, c1_val, c2_val, c3_val,Name,pvaluefile)
    else:
        print("Continous Phenotype!")
        fit_continous_phenotype_on_PRS(traindirec, newtrainfilename, p, 
                        M_val,opt_llk_val,iter_val_val,burn_val,thin_val,r2_val,
                        a_val,c_val,a0k_val,b0k_val,referenceset, p1_val, p2_val, p3_val, c1_val, c2_val, c3_val,Name,pvaluefile)
            
 


 
M_values = [1000]
opt_llk_values = [1]
# opt_llk_values = [2] produced error.

opt_llk_values = [1]
iter_val_values = [1000]
burn_values = [200]
thin_values = [5]
n_threads_values = [1]
r2_values = [0.1]
a_values = [0.1]
c_values = [1]
a0k_values = [0.5]
b0k_values = [0.5]

referencesets = ["custom","EUR_MHC","EUR_noMHC"]
referencesets = [ "EUR_MHC"]
  
result_directory = "SDPR"
# Nested loops to iterate over different parameter values
create_directory(folddirec+os.sep+result_directory)



for p1_val in p_window_size:
 for p2_val in p_slide_size: 
  for p3_val in p_LD_threshold:
   for c1_val in clump_p1:
    for c2_val in clump_r2:
     for c3_val in clump_kb:
      for p in numberofpca:
        for M_val in M_values:
         for opt_llk_val in opt_llk_values:
          for iter_val_val in iter_val_values:
           for burn_val in burn_values:
            for thin_val in thin_values:
              for r2_val in r2_values:
               for a_val in a_values:
                for c_val in c_values:
                 for a0k_val in a0k_values:
                  for b0k_val in b0k_values:
                    for referenceset in referencesets:
                        #referenceset="custom"
                        transform_sdpr_data(folddirec, newtrainfilename, p, 
                        M_val,opt_llk_val,iter_val_val,burn_val,thin_val,r2_val,
                        a_val,c_val,a0k_val,b0k_val,referenceset,
                        str(p1_val), str(p2_val), str(p3_val), str(c1_val), str(c2_val), str(c3_val), result_directory, pvaluefile)

PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC
  --indep-pairwise 200 50 0.25
  --out SampleData1/Fold_0/train_data

63761 MB RAM detected; reserving 31880 MB for main workspace.
491952 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is 0.999894.
491952 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
Pruned 18860 variants from chromosome 1, leaving 20363.
Pruned 19645 variants from chromosome 2, leaving 20067.
Pruned 16414 variants from chromosome 3, leaving 17080.
Pruned 15404 variants from chromosome 4, leaving 16035.
Pruned 14196 variants from chromosome 5, leaving 15379.
Pruned 19368 variants from chromosome 6, leaving 14770.
Pruned 13110 variants from chromosome 7, leaving 13997.
Pruned 12431 variants from chromosome 8, leaving 12966.
Pruned 9982 variants from chromosome 9, leaving 11477.
Pruned 11999 variants from chromosome 10, leaving 12850.
Pruned 12156 variants from chromosome 11, leaving 12221.
Pruned 10979 variants from chromosome 12, leaving 12050.
Pruned 7923 variants from chromosome 13, leaving 9247.
Pruned 7624 variants from chromosome 14, leaving 8448.
Pruned 7387 variants from chromosome 15, leaving 8145.
Pruned 8063 variants from chromosome 16, leaving 8955.
Pruned 7483 variants from chromosome 17, leaving 8361.
Pruned 6767 variants from chromosome 18, leaving 8240.
Pruned 6438 variants from chromosome 19, leaving 6432.
Pruned 5972 variants from chromosome 20, leaving 7202.
Pruned 3426 variants from chromosome 21, leaving 4102.
Pruned 3801 variants from chromosome 22, leaving 4137.
Pruning complete.  239428 of 491952 variants removed.
Marker lists written to SampleData1/Fold_0/train_data.prune.in and
SampleData1/Fold_0/train_data.prune.out .
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC
  --clump SampleData1/SampleData1.txt
  --clump-field P
  --clump-kb 200
  --clump-p1 1
  --clump-r2 0.1
  --clump-snp-field SNP
  --extract SampleData1/Fold_0/train_data.prune.in
  --out SampleData1/Fold_0/train_data

63761 MB RAM detected; reserving 31880 MB for main workspace.
491952 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
--extract: 252524 variants remaining.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is 0.999894.
252524 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--clump: 32781 clumps formed from 33781 top variants.
Results written to SampleData1/Fold_0/train_data.clumped .

Warning: 'rs2394894' is missing from the main dataset, and is a top variant.
Warning: 'rs2074470' is missing from the main dataset, and is a top variant.
Warning: 'rs4980452' is missing from the main dataset, and is a top variant.
2783 more top variant IDs missing; see log file.

PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC
  --extract SampleData1/Fold_0/train_data.valid.snp
  --indep-pairwise 200 50 0.25
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned

63761 MB RAM detected; reserving 31880 MB for main workspace.
491952 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
--extract: 32781 variants remaining.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
32781 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Pruned 0 variants from chromosome 1, leaving 2602.
Pruned 0 variants from chromosome 2, leaving 2410.
Pruned 0 variants from chromosome 3, leaving 2174.
Pruned 0 variants from chromosome 4, leaving 2027.
Pruned 0 variants from chromosome 5, leaving 1973.
Pruned 0 variants from chromosome 6, leaving 1897.
Pruned 0 variants from chromosome 7, leaving 1761.
Pruned 0 variants from chromosome 8, leaving 1629.
Pruned 0 variants from chromosome 9, leaving 1507.
Pruned 0 variants from chromosome 10, leaving 1627.
Pruned 0 variants from chromosome 11, leaving 1551.
Pruned 0 variants from chromosome 12, leaving 1579.
Pruned 0 variants from chromosome 13, leaving 1217.
Pruned 0 variants from chromosome 14, leaving 1061.
Pruned 0 variants from chromosome 15, leaving 1129.
Pruned 0 variants from chromosome 16, leaving 1207.
Pruned 0 variants from chromosome 17, leaving 1152.
Pruned 0 variants from chromosome 18, leaving 1134.
Pruned 0 variants from chromosome 19, leaving 1004.
Pruned 0 variants from chromosome 20, leaving 972.
Pruned 0 variants from chromosome 21, leaving 556.
Pruned 0 variants from chromosome 22, leaving 612.
Pruning complete.  0 of 32781 variants removed.
Marker lists written to
SampleData1/Fold_0/train_data.QC.clumped.pruned.prune.in and
SampleData1/Fold_0/train_data.QC.clumped.pruned.prune.out .
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/test_data.clumped.pruned.log.
Options in effect:
  --bfile SampleData1/Fold_0/test_data
  --extract SampleData1/Fold_0/train_data.valid.snp
  --indep-pairwise 200 50 0.25
  --make-bed
  --out SampleData1/Fold_0/test_data.clumped.pruned

63761 MB RAM detected; reserving 31880 MB for main workspace.
551892 variants loaded from .bim file.
95 people (44 males, 51 females) loaded from .fam.
95 phenotype values loaded from .fam.
--extract: 32781 variants remaining.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 95 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
32781 variants and 95 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/test_data.clumped.pruned.bed +
SampleData1/Fold_0/test_data.clumped.pruned.bim +
SampleData1/Fold_0/test_data.clumped.pruned.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Pruned 5 variants from chromosome 1, leaving 2597.
Pruned 8 variants from chromosome 2, leaving 2402.
Pruned 7 variants from chromosome 3, leaving 2167.
Pruned 8 variants from chromosome 4, leaving 2019.
Pruned 6 variants from chromosome 5, leaving 1967.
Pruned 3 variants from chromosome 6, leaving 1894.
Pruned 6 variants from chromosome 7, leaving 1755.
Pruned 2 variants from chromosome 8, leaving 1627.
Pruned 5 variants from chromosome 9, leaving 1502.
Pruned 2 variants from chromosome 10, leaving 1625.
Pruned 2 variants from chromosome 11, leaving 1549.
Pruned 5 variants from chromosome 12, leaving 1574.
Pruned 2 variants from chromosome 13, leaving 1215.
Pruned 4 variants from chromosome 14, leaving 1057.
Pruned 4 variants from chromosome 15, leaving 1125.
Pruned 3 variants from chromosome 16, leaving 1204.
Pruned 3 variants from chromosome 17, leaving 1149.
Pruned 0 variants from chromosome 18, leaving 1134.
Pruned 1 variant from chromosome 19, leaving 1003.
Pruned 2 variants from chromosome 20, leaving 970.
Pruned 1 variant from chromosome 21, leaving 555.
Pruned 4 variants from chromosome 22, leaving 608.
Pruning complete.  83 of 32781 variants removed.
Marker lists written to SampleData1/Fold_0/test_data.clumped.pruned.prune.in
and SampleData1/Fold_0/test_data.clumped.pruned.prune.out .
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/test_data.log.
Options in effect:
  --bfile SampleData1/Fold_0/test_data.clumped.pruned
  --extract SampleData1/Fold_0/train_data.valid.snp
  --out SampleData1/Fold_0/test_data
  --pca 6

63761 MB RAM detected; reserving 31880 MB for main workspace.
32781 variants loaded from .bim file.
95 people (44 males, 51 females) loaded from .fam.
95 phenotype values loaded from .fam.
--extract: 32781 variants remaining.
Using up to 8 threads (change this with --threads).
Before main variant filters, 95 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
32781 variants and 95 people pass filters and QC.
Phenotype data is quantitative.
Relationship matrix calculation complete.
--pca: Results saved to SampleData1/Fold_0/test_data.eigenval and
SampleData1/Fold_0/test_data.eigenvec .
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --extract SampleData1/Fold_0/train_data.valid.snp
  --out SampleData1/Fold_0/train_data
  --pca 6

63761 MB RAM detected; reserving 31880 MB for main workspace.
32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
--extract: 32781 variants remaining.
Using up to 8 threads (change this with --threads).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
32781 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
Relationship matrix calculation complete.
--pca: Results saved to SampleData1/Fold_0/train_data.eigenval and
SampleData1/Fold_0/train_data.eigenvec .
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.1.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 1
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.1

63761 MB RAM detected; reserving 31880 MB for main workspace.
2602 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
2602 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.1.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.1.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.1.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.

Running SDPR with opt_llk 1
Readed 141 LD blocks.
Readed 90646 SNPs from reference panel
21 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
1534 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.0172132 max beta: 0.0159971
200 iter. h2: 0.017677 max beta: 0.0152579
300 iter. h2: 0.0173169 max beta: 0.0159302
400 iter. h2: 0.0171669 max beta: 0.0143225
500 iter. h2: 0.017415 max beta: 0.0168423
600 iter. h2: 0.0171728 max beta: 0.0122845
700 iter. h2: 0.0168012 max beta: 0.0118124
800 iter. h2: 0.0168011 max beta: 0.0146213
900 iter. h2: 0.0161348 max beta: 0.0117448
1000 iter. h2: 0.0169009 max beta: 0.0138846
h2: 0.0169624 max: 0.0128234
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.2.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 2
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.2

63761 MB RAM detected; reserving 31880 MB for main workspace.
2410 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
2410 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.2.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.2.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.2.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 139 LD blocks.
Readed 92890 SNPs from reference panel
13 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
1383 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.0156064 max beta: 0.0160654
200 iter. h2: 0.0155655 max beta: 0.013046
300 iter. h2: 0.015747 max beta: 0.0128383
400 iter. h2: 0.0159273 max beta: 0.0140063
500 iter. h2: 0.0150668 max beta: 0.0136247
600 iter. h2: 0.0153203 max beta: 0.0113361
700 iter. h2: 0.015464 max beta: 0.0118741
800 iter. h2: 0.0156717 max beta: 0.0133789
900 iter. h2: 0.0153155 max beta: 0.0121683
1000 iter. h2: 0.0152975 max beta: 0.0124437
h2: 0.0154345 max: 0.012398
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.3.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 3
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.3

63761 MB RAM detected; reserving 31880 MB for main workspace.
2174 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
2174 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.3.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.3.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.3.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 119 LD blocks.
Readed 77470 SNPs from reference panel
17 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
1177 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.0128146 max beta: 0.016389
200 iter. h2: 0.0130827 max beta: 0.0161415
300 iter. h2: 0.012487 max beta: 0.012765
400 iter. h2: 0.0133171 max beta: 0.0156502
500 iter. h2: 0.0128646 max beta: 0.0137284
600 iter. h2: 0.0125174 max beta: 0.0150613
700 iter. h2: 0.0133165 max beta: 0.0107014
800 iter. h2: 0.014005 max beta: 0.0131085
900 iter. h2: 0.0132343 max beta: 0.0154957
1000 iter. h2: 0.0132022 max beta: 0.0122708
h2: 0.0130918 max: 0.0139712
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.4.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 4
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.4

63761 MB RAM detected; reserving 31880 MB for main workspace.
2027 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
2027 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.4.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.4.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.4.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 115 LD blocks.
Readed 68848 SNPs from reference panel
11 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
997 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.0108979 max beta: 0.0151651
200 iter. h2: 0.0120693 max beta: 0.0181484
300 iter. h2: 0.0117164 max beta: 0.0139974
400 iter. h2: 0.0113044 max beta: 0.0155865
500 iter. h2: 0.011623 max beta: 0.0136567
600 iter. h2: 0.0120671 max beta: 0.01394
700 iter. h2: 0.0113018 max beta: 0.0180421
800 iter. h2: 0.0107989 max beta: 0.0137218
900 iter. h2: 0.011559 max beta: 0.0139951
1000 iter. h2: 0.0120947 max beta: 0.0168791
h2: 0.0115078 max: 0.0148792
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.5.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 5
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.5

63761 MB RAM detected; reserving 31880 MB for main workspace.
1973 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1973 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.5.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.5.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.5.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 114 LD blocks.

Readed 69616 SNPs from reference panel
10 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
1123 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.0113242 max beta: 0.0101329
200 iter. h2: 0.0112549 max beta: 0.0109094
300 iter. h2: 0.0113853 max beta: 0.010986
400 iter. h2: 0.0109156 max beta: 0.00951889
500 iter. h2: 0.0115522 max beta: 0.0113129
600 iter. h2: 0.0115715 max beta: 0.0102429
700 iter. h2: 0.0111383 max beta: 0.00971955
800 iter. h2: 0.0116266 max beta: 0.0112668
900 iter. h2: 0.0109529 max beta: 0.0110106
1000 iter. h2: 0.0109193 max beta: 0.00831612
h2: 0.0112478 max: 0.0100864
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.6.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 6
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.6

63761 MB RAM detected; reserving 31880 MB for main workspace.
1897 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1897 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.6.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.6.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.6.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 95 LD blocks.
Readed 74093 SNPs from reference panel
9 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
1098 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.0222565 max beta: 0.0241725
200 iter. h2: 0.0213653 max beta: 0.0237193
300 iter. h2: 0.022144 max beta: 0.0246476
400 iter. h2: 0.0218326 max beta: 0.0261976
500 iter. h2: 0.0217807 max beta: 0.0242341
600 iter. h2: 0.021717 max beta: 0.0260209
700 iter. h2: 0.0222466 max beta: 0.0267494
800 iter. h2: 0.0230049 max beta: 0.0261314
900 iter. h2: 0.0217726 max beta: 0.0253364
1000 iter. h2: 0.0215396 max beta: 0.0261504
h2: 0.0216727 max: 0.0241731
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.7.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 7
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.7

63761 MB RAM detected; reserving 31880 MB for main workspace.
1761 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1761 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.7.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.7.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.7.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 93 LD blocks.
Readed 61401 SNPs from reference panel
9 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
984 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.0102583 max beta: 0.0110389
200 iter. h2: 0.0104707 max beta: 0.0104
300 iter. h2: 0.0100047 max beta: 0.0107757
400 iter. h2: 0.0103122 max beta: 0.011093
500 iter. h2: 0.0106266 max beta: 0.0103317
600 iter. h2: 0.00991343 max beta: 0.0101816
700 iter. h2: 0.00933241 max beta: 0.00989029
800 iter. h2: 0.009642 max beta: 0.010025
900 iter. h2: 0.0105689 max beta: 0.00972386
1000 iter. h2: 0.00984209 max beta: 0.00984023
h2: 0.0100229 max: 0.00907835
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.8.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 8
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.8

63761 MB RAM detected; reserving 31880 MB for main workspace.
1629 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1629 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.8.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.8.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.8.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 88 LD blocks.
Readed 60562 SNPs from reference panel
10 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
911 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.0106086 max beta: 0.0107275
200 iter. h2: 0.0103015 max beta: 0.0108155
300 iter. h2: 0.0105966 max beta: 0.0102289
400 iter. h2: 0.0110324 max beta: 0.00965508
500 iter. h2: 0.0107186 max beta: 0.00970525
600 iter. h2: 0.0110791 max beta: 0.0110767
700 iter. h2: 0.0103138 max beta: 0.0116988
800 iter. h2: 0.0106846 max beta: 0.0104615
900 iter. h2: 0.0106561 max beta: 0.0104873
1000 iter. h2: 0.0106486 max beta: 0.0104833
h2: 0.0108219 max: 0.00888498
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.9.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 9
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.9

63761 MB RAM detected; reserving 31880 MB for main workspace.
1507 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1507 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.9.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.9.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.9.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 85 LD blocks.

Readed 51895 SNPs from reference panel
7 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
933 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.0101468 max beta: 0.0143971
200 iter. h2: 0.00940235 max beta: 0.0119349
300 iter. h2: 0.00936168 max beta: 0.0108285
400 iter. h2: 0.00989631 max beta: 0.0130489
500 iter. h2: 0.00930964 max beta: 0.0121526
600 iter. h2: 0.00970508 max beta: 0.0104873
700 iter. h2: 0.00975673 max beta: 0.0104515
800 iter. h2: 0.0101543 max beta: 0.0107434
900 iter. h2: 0.00961735 max beta: 0.0113878
1000 iter. h2: 0.00984427 max beta: 0.00997854
h2: 0.00970007 max: 0.011455
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.10.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 10
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.10

63761 MB RAM detected; reserving 31880 MB for main workspace.
1627 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1627 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.10.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.10.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.10.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 89 LD blocks.
Readed 59192 SNPs from reference panel
8 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
948 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.0100166 max beta: 0.0102835
200 iter. h2: 0.00998731 max beta: 0.00966656
300 iter. h2: 0.00987772 max beta: 0.0115889
400 iter. h2: 0.01056 max beta: 0.0103726
500 iter. h2: 0.00972428 max beta: 0.00987202
600 iter. h2: 0.010032 max beta: 0.012
700 iter. h2: 0.00981794 max beta: 0.0109757
800 iter. h2: 0.00990576 max beta: 0.0103755
900 iter. h2: 0.00971172 max beta: 0.0104994
1000 iter. h2: 0.0102057 max beta: 0.0091045
h2: 0.0098884 max: 0.00930131
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.11.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 11
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.11

63761 MB RAM detected; reserving 31880 MB for main workspace.
1551 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1551 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.11.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.11.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.11.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 92 LD blocks.
Readed 56862 SNPs from reference panel
11 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
910 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.0100643 max beta: 0.0131549
200 iter. h2: 0.00989103 max beta: 0.00983002
300 iter. h2: 0.0108968 max beta: 0.0175444
400 iter. h2: 0.0100021 max beta: 0.0109928
500 iter. h2: 0.0106242 max beta: 0.0122409
600 iter. h2: 0.00999472 max beta: 0.0142912
700 iter. h2: 0.00922337 max beta: 0.0148111
800 iter. h2: 0.0101707 max beta: 0.0127414
900 iter. h2: 0.0101536 max beta: 0.0137017
1000 iter. h2: 0.0103991 max beta: 0.0124138
h2: 0.0101701 max: 0.012036
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.12.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 12
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.12

63761 MB RAM detected; reserving 31880 MB for main workspace.
1579 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1579 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.12.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.12.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.12.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 83 LD blocks.
Readed 53870 SNPs from reference panel
11 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
935 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.00980451 max beta: 0.00979705
200 iter. h2: 0.00975268 max beta: 0.0116816
300 iter. h2: 0.00991241 max beta: 0.0112943
400 iter. h2: 0.00948072 max beta: 0.0096462
500 iter. h2: 0.00976 max beta: 0.0107901
600 iter. h2: 0.0100123 max beta: 0.0105118
700 iter. h2: 0.00963263 max beta: 0.00964335
800 iter. h2: 0.0104568 max beta: 0.0118146
900 iter. h2: 0.0104416 max beta: 0.0145389
1000 iter. h2: 0.00933289 max beta: 0.00988351
h2: 0.0100607 max: 0.00986931
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.13.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 13
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.13

63761 MB RAM detected; reserving 31880 MB for main workspace.
1217 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1217 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.13.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.13.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.13.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 69 LD blocks.
Readed 41857 SNPs from reference panel
8 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
667 common SNPs among reference, validation and gwas summary statistics.

100 iter. h2: 0.00689781 max beta: 0.0136858
200 iter. h2: 0.00666773 max beta: 0.0124508
300 iter. h2: 0.0064374 max beta: 0.0115627
400 iter. h2: 0.00665136 max beta: 0.0131383
500 iter. h2: 0.00700803 max beta: 0.0107324
600 iter. h2: 0.00704394 max beta: 0.0100989
700 iter. h2: 0.00690073 max beta: 0.00955497
800 iter. h2: 0.00702911 max beta: 0.010818
900 iter. h2: 0.00722576 max beta: 0.0100551
1000 iter. h2: 0.00705446 max beta: 0.00944833
h2: 0.00680077 max: 0.0105782
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.14.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 14
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.14

63761 MB RAM detected; reserving 31880 MB for main workspace.
1061 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1061 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.14.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.14.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.14.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 59 LD blocks.
Readed 36507 SNPs from reference panel
8 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
611 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.00686946 max beta: 0.0110164
200 iter. h2: 0.00728609 max beta: 0.00941711
300 iter. h2: 0.00698449 max beta: 0.00929637
400 iter. h2: 0.00687761 max beta: 0.011343
500 iter. h2: 0.00681662 max beta: 0.0125431
600 iter. h2: 0.00669344 max beta: 0.0118503
700 iter. h2: 0.0064605 max beta: 0.0103351
800 iter. h2: 0.00676004 max beta: 0.00964348
900 iter. h2: 0.00680562 max beta: 0.00933407
1000 iter. h2: 0.00684507 max beta: 0.00885872
h2: 0.00686144 max: 0.00852761
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.15.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 15
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.15

63761 MB RAM detected; reserving 31880 MB for main workspace.
1129 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1129 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.15.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.15.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.15.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 60 LD blocks.
Readed 33321 SNPs from reference panel
3 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
643 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.00688405 max beta: 0.0097804
200 iter. h2: 0.00635235 max beta: 0.0100299
300 iter. h2: 0.00676252 max beta: 0.010927
400 iter. h2: 0.00615791 max beta: 0.0101576
500 iter. h2: 0.00653553 max beta: 0.0102644
600 iter. h2: 0.00700472 max beta: 0.00991732
700 iter. h2: 0.00657048 max beta: 0.00919678
800 iter. h2: 0.00629028 max beta: 0.0126535
900 iter. h2: 0.00594907 max beta: 0.00831653
1000 iter. h2: 0.00698713 max beta: 0.0123412
h2: 0.00660461 max: 0.0090013
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.16.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 16
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.16

63761 MB RAM detected; reserving 31880 MB for main workspace.
1207 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1207 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.16.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.16.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.16.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 65 LD blocks.
Readed 34723 SNPs from reference panel
5 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
662 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.0070211 max beta: 0.0101289
200 iter. h2: 0.00632778 max beta: 0.00947934
300 iter. h2: 0.00667403 max beta: 0.00968126
400 iter. h2: 0.00695347 max beta: 0.00880777
500 iter. h2: 0.00697973 max beta: 0.00964156
600 iter. h2: 0.00663877 max beta: 0.00885652
700 iter. h2: 0.00680119 max beta: 0.00968021
800 iter. h2: 0.00624462 max beta: 0.00976108
900 iter. h2: 0.00630077 max beta: 0.00885165
1000 iter. h2: 0.00609114 max beta: 0.00838334
h2: 0.00667158 max: 0.00803696
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.17.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 17
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.17

63761 MB RAM detected; reserving 31880 MB for main workspace.
1152 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1152 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.17.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.17.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.17.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 58 LD blocks.
Readed 29985 SNPs from reference panel

5 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
664 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.00718788 max beta: 0.0119388
200 iter. h2: 0.00713066 max beta: 0.0138855
300 iter. h2: 0.00800624 max beta: 0.0142209
400 iter. h2: 0.00725406 max beta: 0.0159499
500 iter. h2: 0.00742887 max beta: 0.0127837
600 iter. h2: 0.00705555 max beta: 0.0126019
700 iter. h2: 0.00699093 max beta: 0.0111475
800 iter. h2: 0.00791107 max beta: 0.0108898
900 iter. h2: 0.00703141 max beta: 0.0117534
1000 iter. h2: 0.00704938 max beta: 0.0119216
h2: 0.00736895 max: 0.0116472
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.18.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 18
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.18

63761 MB RAM detected; reserving 31880 MB for main workspace.
1134 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1134 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.18.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.18.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.18.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 59 LD blocks.
Readed 32809 SNPs from reference panel
6 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
630 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.006181 max beta: 0.00939778
200 iter. h2: 0.00635929 max beta: 0.0112419
300 iter. h2: 0.00650925 max beta: 0.00964682
400 iter. h2: 0.00565811 max beta: 0.00897583
500 iter. h2: 0.00537371 max beta: 0.00829847
600 iter. h2: 0.00543499 max beta: 0.0107614
700 iter. h2: 0.00622742 max beta: 0.0101966
800 iter. h2: 0.00615524 max beta: 0.00922802
900 iter. h2: 0.0063617 max beta: 0.0108727
1000 iter. h2: 0.00573343 max beta: 0.00808688
h2: 0.00604677 max: 0.00809447
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.19.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 19
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.19

63761 MB RAM detected; reserving 31880 MB for main workspace.
1004 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
1004 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.19.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.19.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.19.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 47 LD blocks.
Readed 20353 SNPs from reference panel
5 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
586 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.00699158 max beta: 0.0140377
200 iter. h2: 0.00686205 max beta: 0.0118305
300 iter. h2: 0.00678759 max beta: 0.0117924
400 iter. h2: 0.0070055 max beta: 0.0140159
500 iter. h2: 0.0064682 max beta: 0.0131325
600 iter. h2: 0.00662487 max beta: 0.0124159
700 iter. h2: 0.00702934 max beta: 0.0131909
800 iter. h2: 0.00670054 max beta: 0.0136054
900 iter. h2: 0.00680132 max beta: 0.0137965
1000 iter. h2: 0.00679404 max beta: 0.0101491
h2: 0.00682871 max: 0.0116555
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.20.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 20
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.20

63761 MB RAM detected; reserving 31880 MB for main workspace.
972 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
972 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.20.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.20.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.20.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 55 LD blocks.
Readed 28792 SNPs from reference panel
4 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
588 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.00569312 max beta: 0.0108362
200 iter. h2: 0.00588683 max beta: 0.0125235
300 iter. h2: 0.0057754 max beta: 0.0093041
400 iter. h2: 0.00566793 max beta: 0.0120514
500 iter. h2: 0.00603489 max beta: 0.0105052
600 iter. h2: 0.00601398 max beta: 0.00983945
700 iter. h2: 0.00536715 max beta: 0.0104365
800 iter. h2: 0.00595145 max beta: 0.0111238
900 iter. h2: 0.00552997 max beta: 0.00856312
1000 iter. h2: 0.00534995 max beta: 0.0097895
h2: 0.00576201 max: 0.00976575
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.21.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 21
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.21

63761 MB RAM detected; reserving 31880 MB for main workspace.
556 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
556 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.21.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.21.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.21.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 36 LD blocks.
Readed 16015 SNPs from reference panel

5 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
309 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.00292272 max beta: 0.00865774
200 iter. h2: 0.00289156 max beta: 0.00784092
300 iter. h2: 0.00331582 max beta: 0.00924418
400 iter. h2: 0.00254689 max beta: 0.00815854
500 iter. h2: 0.00288461 max beta: 0.00811145
600 iter. h2: 0.00291352 max beta: 0.00717188
700 iter. h2: 0.00316228 max beta: 0.0106832
800 iter. h2: 0.00339047 max beta: 0.0112351
900 iter. h2: 0.00280288 max beta: 0.0109824
1000 iter. h2: 0.00290491 max beta: 0.00802953
h2: 0.00287794 max: 0.00752954
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/train_data.QC.clumped.pruned.22.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --chr 22
  --make-bed
  --out SampleData1/Fold_0/train_data.QC.clumped.pruned.22

63761 MB RAM detected; reserving 31880 MB for main workspace.
612 out of 32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
612 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--make-bed to SampleData1/Fold_0/train_data.QC.clumped.pruned.22.bed +
SampleData1/Fold_0/train_data.QC.clumped.pruned.22.bim +
SampleData1/Fold_0/train_data.QC.clumped.pruned.22.fam ... 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899done.
Running SDPR with opt_llk 1
Readed 35 LD blocks.
Readed 15910 SNPs from reference panel
1 SNPs have flipped alleles between summary statistics and reference panel.
0 SNPs removed due to mismatch of allels between summary statistics and reference panel.
369 common SNPs among reference, validation and gwas summary statistics.
100 iter. h2: 0.00489849 max beta: 0.0164828
200 iter. h2: 0.00472272 max beta: 0.0143096
300 iter. h2: 0.00439038 max beta: 0.0116975
400 iter. h2: 0.00508851 max beta: 0.0127566
500 iter. h2: 0.00457888 max beta: 0.0141306
600 iter. h2: 0.00478541 max beta: 0.01185
700 iter. h2: 0.00489391 max beta: 0.0175811
800 iter. h2: 0.00475961 max beta: 0.0143384
900 iter. h2: 0.00488078 max beta: 0.0121491
1000 iter. h2: 0.00525259 max beta: 0.017064
h2: 0.00471013 max: 0.0139565
SampleData1/Fold_0/result/1
SampleData1/Fold_0/result/2
SampleData1/Fold_0/result/3
SampleData1/Fold_0/result/4
SampleData1/Fold_0/result/5
SampleData1/Fold_0/result/6
SampleData1/Fold_0/result/7
SampleData1/Fold_0/result/8
SampleData1/Fold_0/result/9
SampleData1/Fold_0/result/10
SampleData1/Fold_0/result/11
SampleData1/Fold_0/result/12
SampleData1/Fold_0/result/13
SampleData1/Fold_0/result/14
SampleData1/Fold_0/result/15
SampleData1/Fold_0/result/16
SampleData1/Fold_0/result/17
SampleData1/Fold_0/result/18
SampleData1/Fold_0/result/19
SampleData1/Fold_0/result/20
SampleData1/Fold_0/result/21
SampleData1/Fold_0/result/22
          SNP A1      beta
0   rs9442373  C  0.001203
1   rs2494591  A -0.000086
2   rs2494641  T  0.000603
3   rs4648639  G -0.001681
4  rs10797342  C  0.001352
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/SDPR/train_data.log.
Options in effect:
  --bfile SampleData1/Fold_0/train_data.QC.clumped.pruned
  --extract SampleData1/Fold_0/train_data.valid.snp
  --out SampleData1/Fold_0/SDPR/train_data
  --q-score-range SampleData1/Fold_0/range_list SampleData1/Fold_0/SNP.pvalue
  --score SampleData1/Fold_0/train_data.SDPRNew 1 2 3 header

63761 MB RAM detected; reserving 31880 MB for main workspace.
32781 variants loaded from .bim file.
380 people (183 males, 197 females) loaded from .fam.
380 phenotype values loaded from .fam.
--extract: 32781 variants remaining.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 380 founders and 0 nonfounders present.
Calculating allele frequencies... 10111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989 done.
Total genotyping rate is exactly 1.
32781 variants and 380 people pass filters and QC.
Phenotype data is quantitative.
--score: 16754 valid predictors loaded.

Warning: 1908 lines skipped in --score file (1908 due to variant ID mismatch, 0
due to allele code mismatch); see SampleData1/Fold_0/SDPR/train_data.nopred for
details.
Warning: 19814 lines skipped in --q-score-range data file.

--score: 20 ranges processed.
Results written to SampleData1/Fold_0/SDPR/train_data.*.profile.
PLINK v1.90b7.2 64-bit (11 Dec 2023)           www.cog-genomics.org/plink/1.9/
(C) 2005-2023 Shaun Purcell, Christopher Chang   GNU General Public License v3
Logging to SampleData1/Fold_0/SDPR/test_data.log.
Options in effect:
  --bfile SampleData1/Fold_0/test_data.clumped.pruned
  --extract SampleData1/Fold_0/train_data.valid.snp
  --out SampleData1/Fold_0/SDPR/test_data
  --q-score-range SampleData1/Fold_0/range_list SampleData1/Fold_0/SNP.pvalue
  --score SampleData1/Fold_0/train_data.SDPRNew 1 2 3 header

63761 MB RAM detected; reserving 31880 MB for main workspace.
32781 variants loaded from .bim file.
95 people (44 males, 51 females) loaded from .fam.
95 phenotype values loaded from .fam.
--extract: 32781 variants remaining.
Using 1 thread (no multithreaded calculations invoked).
Before main variant filters, 95 founders and 0 nonfounders present.
Calculating allele frequencies... 0%1%2%3%4%5%6%7%8%9%10%11%12%13%14%15%16%17%18%19%20%21%22%23%24%25%26%27%28%29%30%31%32%33%34%35%36%37%38%39%40%41%42%43%44%45%46%47%48%49%50%51%52%53%54%55%56%57%58%59%60%61%62%63%64%65%66%67%68%69%70%71%72%73%74%75%76%77%78%79%80%81%82%83%84%85%86%87%88%89%90%91%92%93%94%95%96%97%98%99% done.
Total genotyping rate is exactly 1.
32781 variants and 95 people pass filters and QC.
Phenotype data is quantitative.
--score: 16754 valid predictors loaded.

Warning: 1908 lines skipped in --score file (1908 due to variant ID mismatch, 0
due to allele code mismatch); see SampleData1/Fold_0/SDPR/test_data.nopred for
details.
Warning: 19814 lines skipped in --q-score-range data file.

--score: 20 ranges processed.
Results written to SampleData1/Fold_0/SDPR/test_data.*.profile.
Continous Phenotype!
1e-10
3.3598182862837877e-10
1.1288378916846883e-09
3.792690190732254e-09
1.274274985703132e-08
4.281332398719396e-08
1.438449888287663e-07
4.832930238571752e-07
1.6237767391887209e-06
5.455594781168514e-06
1.8329807108324338e-05
6.158482110660255e-05
0.00020691380811147902
0.0006951927961775605
0.002335721469090121
0.007847599703514606
0.026366508987303555
0.08858667904100832
0.2976351441631313
1.0

Repeat the process for each fold.

Change the foldnumber variable.

#foldnumber = sys.argv[1]
foldnumber = "0"  # Setting 'foldnumber' to "0"

Or uncomment the following line:

# foldnumber = sys.argv[1]
python SDPR.py 0
python SDPR.py 1
python SDPR.py 2
python SDPR.py 3
python SDPR.py 4

The following files should exist after the execution:

1. SampleData1/Fold_0/SDPR/Results.csv

2. SampleData1/Fold_1/SDPR/Results.csv

3. SampleData1/Fold_2/SDPR/Results.csv

4. SampleData1/Fold_3/SDPR/Results.csv

5. SampleData1/Fold_4/SDPR/Results.csv

Check the results file for each fold.

import os
 
# List of file names to check for existence
f = [
    "./"+filedirec+"/Fold_0"+os.sep+result_directory+"Results.csv",
    "./"+filedirec+"/Fold_1"+os.sep+result_directory+"Results.csv",
    "./"+filedirec+"/Fold_2"+os.sep+result_directory+"Results.csv",
    "./"+filedirec+"/Fold_3"+os.sep+result_directory+"Results.csv",
    "./"+filedirec+"/Fold_4"+os.sep+result_directory+"Results.csv",
]


# Loop through each file name in the list
for loop in range(0,5):
    # Check if the file exists in the specified directory for the given fold
    if os.path.exists(filedirec+os.sep+"Fold_"+str(loop)+os.sep+result_directory+os.sep+"Results.csv"):
        temp = pd.read_csv(filedirec+os.sep+"Fold_"+str(loop)+os.sep+result_directory+os.sep+"Results.csv")
        print("Fold_",loop, "Yes, the file exists.")
        #print(temp.head())
        print("Number of P-values processed: ",len(temp))
        # Print a message indicating that the file exists
    
    else:
        # Print a message indicating that the file does not exist
        print("Fold_",loop, "No, the file does not exist.")

Fold_ 0 Yes, the file exists.
Number of P-values processed:  60
Fold_ 1 Yes, the file exists.
Number of P-values processed:  60
Fold_ 2 Yes, the file exists.
Number of P-values processed:  60
Fold_ 3 Yes, the file exists.
Number of P-values processed:  60
Fold_ 4 Yes, the file exists.
Number of P-values processed:  60

Sum the results for each fold.

print("We have to ensure when we sum the entries across all Folds, the same rows are merged!")

def sum_and_average_columns(data_frames):
    """Sum and average numerical columns across multiple DataFrames, and keep non-numerical columns unchanged."""
    # Initialize DataFrame to store the summed results for numerical columns
    summed_df = pd.DataFrame()
    non_numerical_df = pd.DataFrame()
    
    for df in data_frames:
        # Identify numerical and non-numerical columns
        numerical_cols = df.select_dtypes(include=[np.number]).columns
        non_numerical_cols = df.select_dtypes(exclude=[np.number]).columns
        
        # Sum numerical columns
        if summed_df.empty:
            summed_df = pd.DataFrame(0, index=range(len(df)), columns=numerical_cols)
        
        summed_df[numerical_cols] = summed_df[numerical_cols].add(df[numerical_cols], fill_value=0)
        
        # Keep non-numerical columns (take the first non-numerical entry for each column)
        if non_numerical_df.empty:
            non_numerical_df = df[non_numerical_cols]
        else:
            non_numerical_df[non_numerical_cols] = non_numerical_df[non_numerical_cols].combine_first(df[non_numerical_cols])
    
    # Divide the summed values by the number of dataframes to get the average
    averaged_df = summed_df / len(data_frames)
    
    # Combine numerical and non-numerical DataFrames
    result_df = pd.concat([averaged_df, non_numerical_df], axis=1)
    
    return result_df

from functools import reduce

import os
import pandas as pd
from functools import reduce

def find_common_rows(allfoldsframe):
    # Define the performance columns that need to be excluded
    performance_columns = [
        'Train_null_model', 'Train_pure_prs', 'Train_best_model',
        'Test_pure_prs', 'Test_null_model', 'Test_best_model'
    ]
    important_columns = [
        'clump_p1',
        'clump_r2',
        'clump_kb',
        'p_window_size',
        'p_slide_size',
        'p_LD_threshold',
        'pvalue',
        'referencepanel',
   

        "SDPR_M_val",
        "SDPR_opt_llk_val",
        "SDPR_iter_val_val",
        "SDPR_burn_val",
        "SDPR_thin_val",
        "SDPR_r2_val",
        "SDPR_a_val",
        "SDPR_c_val",
        "SDPR_a0k_val",
        "SDPR_b0k_val",
        "SDPR_referenceset",    
        
        
        'window_shift',
        'numberofpca',
        'tempalpha',
        'l1weight',
         
       
    ]
    # Function to remove performance columns from a DataFrame
    def drop_performance_columns(df):
        return df.drop(columns=performance_columns, errors='ignore')
    
    def get_important_columns(df ):
        existing_columns = [col for col in important_columns if col in df.columns]
        if existing_columns:
            return df[existing_columns].copy()
        else:
            return pd.DataFrame()

    # Drop performance columns from all DataFrames in the list
    allfoldsframe_dropped = [drop_performance_columns(df) for df in allfoldsframe]
    
    # Get the important columns.
    allfoldsframe_dropped = [get_important_columns(df) for df in allfoldsframe_dropped]    
    
    # Iteratively find common rows and track unique and common rows
    common_rows = allfoldsframe_dropped[0]
    for i in range(1, len(allfoldsframe_dropped)):
        # Get the next DataFrame
        next_df = allfoldsframe_dropped[i]

        # Count unique rows in the current DataFrame and the next DataFrame
        unique_in_common = common_rows.shape[0]
        unique_in_next = next_df.shape[0]

        # Find common rows between the current common_rows and the next DataFrame
        common_rows = pd.merge(common_rows, next_df, how='inner')
    
        # Count the common rows after merging
        common_count = common_rows.shape[0]

        # Print the unique and common row counts
        print(f"Iteration {i}:")
        print(f"Unique rows in current common DataFrame: {unique_in_common}")
        print(f"Unique rows in next DataFrame: {unique_in_next}")
        print(f"Common rows after merge: {common_count}\n")
    # Now that we have the common rows, extract these from the original DataFrames
 
    extracted_common_rows_frames = []
    for original_df in allfoldsframe:
        # Merge the common rows with the original DataFrame, keeping only the rows that match the common rows
        extracted_common_rows = pd.merge(common_rows, original_df, how='inner', on=common_rows.columns.tolist())
        
        # Add the DataFrame with the extracted common rows to the list
        extracted_common_rows_frames.append(extracted_common_rows)

    # Print the number of rows in the common DataFrames
    for i, df in enumerate(extracted_common_rows_frames):
        print(f"DataFrame {i + 1} with extracted common rows has {df.shape[0]} rows.")

    # Return the list of DataFrames with extracted common rows
    return extracted_common_rows_frames



# Example usage (assuming allfoldsframe is populated as shown earlier):
allfoldsframe = []

# Loop through each file name in the list
for loop in range(0, 5):
    # Check if the file exists in the specified directory for the given fold
    file_path = os.path.join(filedirec, "Fold_" + str(loop), result_directory, "Results.csv")
    if os.path.exists(file_path):
        allfoldsframe.append(pd.read_csv(file_path))
        # Print a message indicating that the file exists
        print("Fold_", loop, "Yes, the file exists.")
    else:
        # Print a message indicating that the file does not exist
        print("Fold_", loop, "No, the file does not exist.")

# Find the common rows across all folds and return the list of extracted common rows
extracted_common_rows_list = find_common_rows(allfoldsframe)
 
# Sum the values column-wise
# For string values, do not sum it the values are going to be the same for each fold.
# Only sum the numeric values.

divided_result = sum_and_average_columns(extracted_common_rows_list)
  
print(divided_result)

 

We have to ensure when we sum the entries across all Folds, the same rows are merged!
Fold_ 0 Yes, the file exists.
Fold_ 1 Yes, the file exists.
Fold_ 2 Yes, the file exists.
Fold_ 3 Yes, the file exists.
Fold_ 4 Yes, the file exists.
Iteration 1:
Unique rows in current common DataFrame: 60
Unique rows in next DataFrame: 60
Common rows after merge: 60

Iteration 2:
Unique rows in current common DataFrame: 60
Unique rows in next DataFrame: 60
Common rows after merge: 60

Iteration 3:
Unique rows in current common DataFrame: 60
Unique rows in next DataFrame: 60
Common rows after merge: 60

Iteration 4:
Unique rows in current common DataFrame: 60
Unique rows in next DataFrame: 60
Common rows after merge: 60

DataFrame 1 with extracted common rows has 60 rows.
DataFrame 2 with extracted common rows has 60 rows.
DataFrame 3 with extracted common rows has 60 rows.
DataFrame 4 with extracted common rows has 60 rows.
DataFrame 5 with extracted common rows has 60 rows.

/tmp/ipykernel_315843/3744722845.py:24: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  non_numerical_df[non_numerical_cols] = non_numerical_df[non_numerical_cols].combine_first(df[non_numerical_cols])
/tmp/ipykernel_315843/3744722845.py:24: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  non_numerical_df[non_numerical_cols] = non_numerical_df[non_numerical_cols].combine_first(df[non_numerical_cols])
/tmp/ipykernel_315843/3744722845.py:24: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  non_numerical_df[non_numerical_cols] = non_numerical_df[non_numerical_cols].combine_first(df[non_numerical_cols])
/tmp/ipykernel_315843/3744722845.py:24: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  non_numerical_df[non_numerical_cols] = non_numerical_df[non_numerical_cols].combine_first(df[non_numerical_cols])

    clump_p1  clump_r2  clump_kb  p_window_size  p_slide_size  p_LD_threshold  \
0        1.0       0.1     200.0          200.0          50.0            0.25   
1        1.0       0.1     200.0          200.0          50.0            0.25   
2        1.0       0.1     200.0          200.0          50.0            0.25   
3        1.0       0.1     200.0          200.0          50.0            0.25   
4        1.0       0.1     200.0          200.0          50.0            0.25   
5        1.0       0.1     200.0          200.0          50.0            0.25   
6        1.0       0.1     200.0          200.0          50.0            0.25   
7        1.0       0.1     200.0          200.0          50.0            0.25   
8        1.0       0.1     200.0          200.0          50.0            0.25   
9        1.0       0.1     200.0          200.0          50.0            0.25   
10       1.0       0.1     200.0          200.0          50.0            0.25   
11       1.0       0.1     200.0          200.0          50.0            0.25   
12       1.0       0.1     200.0          200.0          50.0            0.25   
13       1.0       0.1     200.0          200.0          50.0            0.25   
14       1.0       0.1     200.0          200.0          50.0            0.25   
15       1.0       0.1     200.0          200.0          50.0            0.25   
16       1.0       0.1     200.0          200.0          50.0            0.25   
17       1.0       0.1     200.0          200.0          50.0            0.25   
18       1.0       0.1     200.0          200.0          50.0            0.25   
19       1.0       0.1     200.0          200.0          50.0            0.25   
20       1.0       0.1     200.0          200.0          50.0            0.25   
21       1.0       0.1     200.0          200.0          50.0            0.25   
22       1.0       0.1     200.0          200.0          50.0            0.25   
23       1.0       0.1     200.0          200.0          50.0            0.25   
24       1.0       0.1     200.0          200.0          50.0            0.25   
25       1.0       0.1     200.0          200.0          50.0            0.25   
26       1.0       0.1     200.0          200.0          50.0            0.25   
27       1.0       0.1     200.0          200.0          50.0            0.25   
28       1.0       0.1     200.0          200.0          50.0            0.25   
29       1.0       0.1     200.0          200.0          50.0            0.25   
30       1.0       0.1     200.0          200.0          50.0            0.25   
31       1.0       0.1     200.0          200.0          50.0            0.25   
32       1.0       0.1     200.0          200.0          50.0            0.25   
33       1.0       0.1     200.0          200.0          50.0            0.25   
34       1.0       0.1     200.0          200.0          50.0            0.25   
35       1.0       0.1     200.0          200.0          50.0            0.25   
36       1.0       0.1     200.0          200.0          50.0            0.25   
37       1.0       0.1     200.0          200.0          50.0            0.25   
38       1.0       0.1     200.0          200.0          50.0            0.25   
39       1.0       0.1     200.0          200.0          50.0            0.25   
40       1.0       0.1     200.0          200.0          50.0            0.25   
41       1.0       0.1     200.0          200.0          50.0            0.25   
42       1.0       0.1     200.0          200.0          50.0            0.25   
43       1.0       0.1     200.0          200.0          50.0            0.25   
44       1.0       0.1     200.0          200.0          50.0            0.25   
45       1.0       0.1     200.0          200.0          50.0            0.25   
46       1.0       0.1     200.0          200.0          50.0            0.25   
47       1.0       0.1     200.0          200.0          50.0            0.25   
48       1.0       0.1     200.0          200.0          50.0            0.25   
49       1.0       0.1     200.0          200.0          50.0            0.25   
50       1.0       0.1     200.0          200.0          50.0            0.25   
51       1.0       0.1     200.0          200.0          50.0            0.25   
52       1.0       0.1     200.0          200.0          50.0            0.25   
53       1.0       0.1     200.0          200.0          50.0            0.25   
54       1.0       0.1     200.0          200.0          50.0            0.25   
55       1.0       0.1     200.0          200.0          50.0            0.25   
56       1.0       0.1     200.0          200.0          50.0            0.25   
57       1.0       0.1     200.0          200.0          50.0            0.25   
58       1.0       0.1     200.0          200.0          50.0            0.25   
59       1.0       0.1     200.0          200.0          50.0            0.25   

          pvalue  SDPR_M_val  SDPR_opt_llk_val  SDPR_iter_val_val  ...  \
0   1.000000e-10      1000.0               1.0             1000.0  ...   
1   3.359818e-10      1000.0               1.0             1000.0  ...   
2   1.128838e-09      1000.0               1.0             1000.0  ...   
3   3.792690e-09      1000.0               1.0             1000.0  ...   
4   1.274275e-08      1000.0               1.0             1000.0  ...   
5   4.281332e-08      1000.0               1.0             1000.0  ...   
6   1.438450e-07      1000.0               1.0             1000.0  ...   
7   4.832930e-07      1000.0               1.0             1000.0  ...   
8   1.623777e-06      1000.0               1.0             1000.0  ...   
9   5.455595e-06      1000.0               1.0             1000.0  ...   
10  1.832981e-05      1000.0               1.0             1000.0  ...   
11  6.158482e-05      1000.0               1.0             1000.0  ...   
12  2.069138e-04      1000.0               1.0             1000.0  ...   
13  6.951928e-04      1000.0               1.0             1000.0  ...   
14  2.335721e-03      1000.0               1.0             1000.0  ...   
15  7.847600e-03      1000.0               1.0             1000.0  ...   
16  2.636651e-02      1000.0               1.0             1000.0  ...   
17  8.858668e-02      1000.0               1.0             1000.0  ...   
18  2.976351e-01      1000.0               1.0             1000.0  ...   
19  1.000000e+00      1000.0               1.0             1000.0  ...   
20  1.000000e-10      1000.0               1.0             1000.0  ...   
21  3.359818e-10      1000.0               1.0             1000.0  ...   
22  1.128838e-09      1000.0               1.0             1000.0  ...   
23  3.792690e-09      1000.0               1.0             1000.0  ...   
24  1.274275e-08      1000.0               1.0             1000.0  ...   
25  4.281332e-08      1000.0               1.0             1000.0  ...   
26  1.438450e-07      1000.0               1.0             1000.0  ...   
27  4.832930e-07      1000.0               1.0             1000.0  ...   
28  1.623777e-06      1000.0               1.0             1000.0  ...   
29  5.455595e-06      1000.0               1.0             1000.0  ...   
30  1.832981e-05      1000.0               1.0             1000.0  ...   
31  6.158482e-05      1000.0               1.0             1000.0  ...   
32  2.069138e-04      1000.0               1.0             1000.0  ...   
33  6.951928e-04      1000.0               1.0             1000.0  ...   
34  2.335721e-03      1000.0               1.0             1000.0  ...   
35  7.847600e-03      1000.0               1.0             1000.0  ...   
36  2.636651e-02      1000.0               1.0             1000.0  ...   
37  8.858668e-02      1000.0               1.0             1000.0  ...   
38  2.976351e-01      1000.0               1.0             1000.0  ...   
39  1.000000e+00      1000.0               1.0             1000.0  ...   
40  1.000000e-10      1000.0               1.0             1000.0  ...   
41  3.359818e-10      1000.0               1.0             1000.0  ...   
42  1.128838e-09      1000.0               1.0             1000.0  ...   
43  3.792690e-09      1000.0               1.0             1000.0  ...   
44  1.274275e-08      1000.0               1.0             1000.0  ...   
45  4.281332e-08      1000.0               1.0             1000.0  ...   
46  1.438450e-07      1000.0               1.0             1000.0  ...   
47  4.832930e-07      1000.0               1.0             1000.0  ...   
48  1.623777e-06      1000.0               1.0             1000.0  ...   
49  5.455595e-06      1000.0               1.0             1000.0  ...   
50  1.832981e-05      1000.0               1.0             1000.0  ...   
51  6.158482e-05      1000.0               1.0             1000.0  ...   
52  2.069138e-04      1000.0               1.0             1000.0  ...   
53  6.951928e-04      1000.0               1.0             1000.0  ...   
54  2.335721e-03      1000.0               1.0             1000.0  ...   
55  7.847600e-03      1000.0               1.0             1000.0  ...   
56  2.636651e-02      1000.0               1.0             1000.0  ...   
57  8.858668e-02      1000.0               1.0             1000.0  ...   
58  2.976351e-01      1000.0               1.0             1000.0  ...   
59  1.000000e+00      1000.0               1.0             1000.0  ...   

    numberofpca  tempalpha  l1weight  Train_pure_prs  Train_null_model  \
0           6.0        0.1       0.1        0.000034            0.2339   
1           6.0        0.1       0.1        0.000033            0.2339   
2           6.0        0.1       0.1        0.000043            0.2339   
3           6.0        0.1       0.1        0.000051            0.2339   
4           6.0        0.1       0.1        0.000052            0.2339   
5           6.0        0.1       0.1        0.000045            0.2339   
6           6.0        0.1       0.1        0.000041            0.2339   
7           6.0        0.1       0.1        0.000035            0.2339   
8           6.0        0.1       0.1        0.000031            0.2339   
9           6.0        0.1       0.1        0.000028            0.2339   
10          6.0        0.1       0.1        0.000026            0.2339   
11          6.0        0.1       0.1        0.000021            0.2339   
12          6.0        0.1       0.1        0.000019            0.2339   
13          6.0        0.1       0.1        0.000018            0.2339   
14          6.0        0.1       0.1        0.000017            0.2339   
15          6.0        0.1       0.1        0.000015            0.2339   
16          6.0        0.1       0.1        0.000011            0.2339   
17          6.0        0.1       0.1        0.000009            0.2339   
18          6.0        0.1       0.1        0.000006            0.2339   
19          6.0        0.1       0.1        0.000004            0.2339   
20          6.0        0.1       0.1        0.000032            0.2339   
21          6.0        0.1       0.1        0.000033            0.2339   
22          6.0        0.1       0.1        0.000037            0.2339   
23          6.0        0.1       0.1        0.000046            0.2339   
24          6.0        0.1       0.1        0.000053            0.2339   
25          6.0        0.1       0.1        0.000045            0.2339   
26          6.0        0.1       0.1        0.000043            0.2339   
27          6.0        0.1       0.1        0.000041            0.2339   
28          6.0        0.1       0.1        0.000039            0.2339   
29          6.0        0.1       0.1        0.000037            0.2339   
30          6.0        0.1       0.1        0.000035            0.2339   
31          6.0        0.1       0.1        0.000028            0.2339   
32          6.0        0.1       0.1        0.000024            0.2339   
33          6.0        0.1       0.1        0.000022            0.2339   
34          6.0        0.1       0.1        0.000020            0.2339   
35          6.0        0.1       0.1        0.000016            0.2339   
36          6.0        0.1       0.1        0.000012            0.2339   
37          6.0        0.1       0.1        0.000010            0.2339   
38          6.0        0.1       0.1        0.000007            0.2339   
39          6.0        0.1       0.1        0.000005            0.2339   
40          6.0        0.1       0.1        0.000027            0.2339   
41          6.0        0.1       0.1        0.000025            0.2339   
42          6.0        0.1       0.1        0.000030            0.2339   
43          6.0        0.1       0.1        0.000045            0.2339   
44          6.0        0.1       0.1        0.000051            0.2339   
45          6.0        0.1       0.1        0.000042            0.2339   
46          6.0        0.1       0.1        0.000041            0.2339   
47          6.0        0.1       0.1        0.000042            0.2339   
48          6.0        0.1       0.1        0.000041            0.2339   
49          6.0        0.1       0.1        0.000038            0.2339   
50          6.0        0.1       0.1        0.000034            0.2339   
51          6.0        0.1       0.1        0.000026            0.2339   
52          6.0        0.1       0.1        0.000022            0.2339   
53          6.0        0.1       0.1        0.000020            0.2339   
54          6.0        0.1       0.1        0.000019            0.2339   
55          6.0        0.1       0.1        0.000015            0.2339   
56          6.0        0.1       0.1        0.000012            0.2339   
57          6.0        0.1       0.1        0.000010            0.2339   
58          6.0        0.1       0.1        0.000007            0.2339   
59          6.0        0.1       0.1        0.000005            0.2339   

    Train_best_model  Test_pure_prs  Test_null_model  Test_best_model  \
0           0.238807   3.087816e-05         0.167588         0.168432   
1           0.239891   3.578328e-05         0.167588         0.168505   
2           0.243775   4.632280e-05         0.167588         0.176662   
3           0.248771   5.394390e-05         0.167588         0.183062   
4           0.254366   5.255336e-05         0.167588         0.189599   
5           0.256841   4.455115e-05         0.167588         0.192057   
6           0.262240   3.993786e-05         0.167588         0.192836   
7           0.263711   3.523723e-05         0.167588         0.198981   
8           0.267340   3.145683e-05         0.167588         0.199955   
9           0.269402   2.943984e-05         0.167588         0.202840   
10          0.276964   2.819259e-05         0.167588         0.215343   
11          0.277325   2.276820e-05         0.167588         0.210744   
12          0.279337   1.961827e-05         0.167588         0.211256   
13          0.296577   1.919340e-05         0.167588         0.236460   
14          0.322650   1.831174e-05         0.167588         0.262383   
15          0.332907   1.535794e-05         0.167588         0.266817   
16          0.334839   1.191532e-05         0.167588         0.269258   
17          0.354087   9.776417e-06         0.167588         0.294173   
18          0.355527   7.081606e-06         0.167588         0.300999   
19          0.364143   5.093186e-06         0.167588         0.312528   
20          0.238910   1.136817e-05         0.167588         0.163354   
21          0.239833   1.913099e-05         0.167588         0.164607   
22          0.242229   2.815483e-05         0.167588         0.170371   
23          0.246685   3.885794e-05         0.167588         0.169926   
24          0.251962   4.469712e-05         0.167588         0.174881   
25          0.251523   3.727720e-05         0.167588         0.173773   
26          0.253152   3.779990e-05         0.167588         0.174003   
27          0.255767   3.836516e-05         0.167588         0.178103   
28          0.257773   3.658559e-05         0.167588         0.176388   
29          0.260652   3.636174e-05         0.167588         0.181648   
30          0.271041   3.583020e-05         0.167588         0.189078   
31          0.270336   2.763527e-05         0.167588         0.189037   
32          0.273020   2.351364e-05         0.167588         0.188700   
33          0.281957   2.153165e-05         0.167588         0.203014   
34          0.298609   2.028784e-05         0.167588         0.218609   
35          0.298459   1.608518e-05         0.167588         0.218626   
36          0.306332   1.240961e-05         0.167588         0.225733   
37          0.317063   1.024690e-05         0.167588         0.236386   
38          0.325801   7.291886e-06         0.167588         0.246015   
39          0.329391   5.165602e-06         0.167588         0.252576   
40          0.235033   6.028943e-07         0.167588         0.170150   
41          0.235276   8.303566e-06         0.167588         0.170800   
42          0.236756   1.779857e-05         0.167588         0.173966   
43          0.240277   3.228089e-05         0.167588         0.176964   
44          0.244669   3.822950e-05         0.167588         0.181533   
45          0.244010   3.129378e-05         0.167588         0.177256   
46          0.245674   3.315658e-05         0.167588         0.177394   
47          0.251100   3.808855e-05         0.167588         0.184544   
48          0.253074   3.740443e-05         0.167588         0.182150   
49          0.255356   3.649169e-05         0.167588         0.186509   
50          0.261798   3.414775e-05         0.167588         0.190045   
51          0.259143   2.554235e-05         0.167588         0.189038   
52          0.261289   2.144253e-05         0.167588         0.187486   
53          0.267670   2.028016e-05         0.167588         0.199308   
54          0.282779   1.930578e-05         0.167588         0.212552   
55          0.283157   1.521589e-05         0.167588         0.213022   
56          0.292747   1.194733e-05         0.167588         0.221919   
57          0.300979   9.763253e-06         0.167588         0.228557   
58          0.308405   6.882028e-06         0.167588         0.235287   
59          0.314027   4.937890e-06         0.167588         0.241164   

    SDPR_referenceset  
0              custom  
1              custom  
2              custom  
3              custom  
4              custom  
5              custom  
6              custom  
7              custom  
8              custom  
9              custom  
10             custom  
11             custom  
12             custom  
13             custom  
14             custom  
15             custom  
16             custom  
17             custom  
18             custom  
19             custom  
20            EUR_MHC  
21            EUR_MHC  
22            EUR_MHC  
23            EUR_MHC  
24            EUR_MHC  
25            EUR_MHC  
26            EUR_MHC  
27            EUR_MHC  
28            EUR_MHC  
29            EUR_MHC  
30            EUR_MHC  
31            EUR_MHC  
32            EUR_MHC  
33            EUR_MHC  
34            EUR_MHC  
35            EUR_MHC  
36            EUR_MHC  
37            EUR_MHC  
38            EUR_MHC  
39            EUR_MHC  
40          EUR_noMHC  
41          EUR_noMHC  
42          EUR_noMHC  
43          EUR_noMHC  
44          EUR_noMHC  
45          EUR_noMHC  
46          EUR_noMHC  
47          EUR_noMHC  
48          EUR_noMHC  
49          EUR_noMHC  
50          EUR_noMHC  
51          EUR_noMHC  
52          EUR_noMHC  
53          EUR_noMHC  
54          EUR_noMHC  
55          EUR_noMHC  
56          EUR_noMHC  
57          EUR_noMHC  
58          EUR_noMHC  
59          EUR_noMHC  

[60 rows x 27 columns]

Results

1. Reporting Based on Best Training Performance:

• One can report the results based on the best performance of the training data. For example, if for a specific combination of hyperparameters, the training performance is high, report the corresponding test performance.

• Example code:

  df = divided_result.sort_values(by='Train_best_model', ascending=False)
  print(df.iloc[0].to_markdown())
  

Binary Phenotypes Result Analysis

You can find the performance quality for binary phenotype using the following template:

This figure shows the 8 different scenarios that can exist in the results, and the following table explains each scenario.

We classified performance based on the following table:

Performance Level	Range
Low Performance	0 to 0.5
Moderate Performance	0.6 to 0.7
High Performance	0.8 to 1

You can match the performance based on the following scenarios:

Scenario	What’s Happening	Implication
High Test, High Train	The model performs well on both training and test datasets, effectively learning the underlying patterns.	The model is well-tuned, generalizes well, and makes accurate predictions on both datasets.
High Test, Moderate Train	The model generalizes well but may not be fully optimized on training data, missing some underlying patterns.	The model is fairly robust but may benefit from further tuning or more training to improve its learning.
High Test, Low Train	An unusual scenario, potentially indicating data leakage or overestimation of test performance.	The model’s performance is likely unreliable; investigate potential data issues or random noise.
Moderate Test, High Train	The model fits the training data well but doesn’t generalize as effectively, capturing only some test patterns.	The model is slightly overfitting; adjustments may be needed to improve generalization on unseen data.
Moderate Test, Moderate Train	The model shows balanced but moderate performance on both datasets, capturing some patterns but missing others.	The model is moderately fitting; further improvements could be made in both training and generalization.
Moderate Test, Low Train	The model underperforms on training data and doesn’t generalize well, leading to moderate test performance.	The model may need more complexity, additional features, or better training to improve on both datasets.
Low Test, High Train	The model overfits the training data, performing poorly on the test set.	The model doesn’t generalize well; simplifying the model or using regularization may help reduce overfitting.
Low Test, Low Train	The model performs poorly on both training and test datasets, failing to learn the data patterns effectively.	The model is underfitting; it may need more complexity, additional features, or more data to improve performance.

Recommendations for Publishing Results

When publishing results, scenarios with moderate train and moderate test performance can be used for complex phenotypes or diseases. However, results showing high train and moderate test, high train and high test, and moderate train and high test are recommended.

For most phenotypes, results typically fall in the moderate train and moderate test performance category.

Continuous Phenotypes Result Analysis

You can find the performance quality for continuous phenotypes using the following template:

This figure shows the 8 different scenarios that can exist in the results, and the following table explains each scenario.

We classified performance based on the following table:

Performance Level	Range
Low Performance	0 to 0.2
Moderate Performance	0.3 to 0.7
High Performance	0.8 to 1

You can match the performance based on the following scenarios:

Scenario	What’s Happening	Implication
High Test, High Train	The model performs well on both training and test datasets, effectively learning the underlying patterns.	The model is well-tuned, generalizes well, and makes accurate predictions on both datasets.
High Test, Moderate Train	The model generalizes well but may not be fully optimized on training data, missing some underlying patterns.	The model is fairly robust but may benefit from further tuning or more training to improve its learning.
High Test, Low Train	An unusual scenario, potentially indicating data leakage or overestimation of test performance.	The model’s performance is likely unreliable; investigate potential data issues or random noise.
Moderate Test, High Train	The model fits the training data well but doesn’t generalize as effectively, capturing only some test patterns.	The model is slightly overfitting; adjustments may be needed to improve generalization on unseen data.
Moderate Test, Moderate Train	The model shows balanced but moderate performance on both datasets, capturing some patterns but missing others.	The model is moderately fitting; further improvements could be made in both training and generalization.
Moderate Test, Low Train	The model underperforms on training data and doesn’t generalize well, leading to moderate test performance.	The model may need more complexity, additional features, or better training to improve on both datasets.
Low Test, High Train	The model overfits the training data, performing poorly on the test set.	The model doesn’t generalize well; simplifying the model or using regularization may help reduce overfitting.
Low Test, Low Train	The model performs poorly on both training and test datasets, failing to learn the data patterns effectively.	The model is underfitting; it may need more complexity, additional features, or more data to improve performance.

Recommendations for Publishing Results

When publishing results, scenarios with moderate train and moderate test performance can be used for complex phenotypes or diseases. However, results showing high train and moderate test, high train and high test, and moderate train and high test are recommended.

For most continuous phenotypes, results typically fall in the moderate train and moderate test performance category.

2. Reporting Generalized Performance:

• One can also report the generalized performance by calculating the difference between the training and test performance, and the sum of the test and training performance. Report the result or hyperparameter combination for which the sum is high and the difference is minimal.

• Example code:

  df = divided_result.copy()
  df['Difference'] = abs(df['Train_best_model'] - df['Test_best_model'])
  df['Sum'] = df['Train_best_model'] + df['Test_best_model']
  
  sorted_df = df.sort_values(by=['Sum', 'Difference'], ascending=[False, True])
  print(sorted_df.iloc[0].to_markdown())
  

3. Reporting Hyperparameters Affecting Test and Train Performance:

• Find the hyperparameters that have more than one unique value and calculate their correlation with the following columns to understand how they are affecting the performance of train and test sets:

  • Train_null_model

  • Train_pure_prs

  • Train_best_model

  • Test_pure_prs

  • Test_null_model

  • Test_best_model

4. Other Analysis

1. Once you have the results, you can find how hyperparameters affect the model performance.

2. Analysis, like overfitting and underfitting, can be performed as well.

3. The way you are going to report the results can vary.

4. Results can be visualized, and other patterns in the data can be explored.

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib notebook

import matplotlib
import numpy as np
import matplotlib.pyplot as plt

df = divided_result.sort_values(by='Train_best_model', ascending=False)
print("1. Reporting Based on Best Training Performance:\n")
print(df.iloc[0].to_markdown())


 
df = divided_result.copy()

# Plot Train and Test best models against p-values
plt.figure(figsize=(10, 6))
plt.plot(df['pvalue'], df['Train_best_model'], label='Train_best_model', marker='o', color='royalblue')
plt.plot(df['pvalue'], df['Test_best_model'], label='Test_best_model', marker='o', color='darkorange')

# Highlight the p-value where both train and test are high
best_index = df[['Train_best_model']].sum(axis=1).idxmax()
best_pvalue = df.loc[best_index, 'pvalue']
best_train = df.loc[best_index, 'Train_best_model']
best_test = df.loc[best_index, 'Test_best_model']

# Use dark colors for the circles
plt.scatter(best_pvalue, best_train, color='darkred', s=100, label=f'Best Performance (Train)', edgecolor='black', zorder=5)
plt.scatter(best_pvalue, best_test, color='darkblue', s=100, label=f'Best Performance (Test)', edgecolor='black', zorder=5)

# Annotate the best performance with p-value, train, and test values
plt.text(best_pvalue, best_train, f'p={best_pvalue:.4g}\nTrain={best_train:.4g}', ha='right', va='bottom', fontsize=9, color='darkred')
plt.text(best_pvalue, best_test, f'p={best_pvalue:.4g}\nTest={best_test:.4g}', ha='right', va='top', fontsize=9, color='darkblue')

# Calculate Difference and Sum
df['Difference'] = abs(df['Train_best_model'] - df['Test_best_model'])
df['Sum'] = df['Train_best_model'] + df['Test_best_model']

# Sort the DataFrame
sorted_df = df.sort_values(by=['Sum', 'Difference'], ascending=[False, True])
#sorted_df = df.sort_values(by=[ 'Difference','Sum'], ascending=[  True,False])

# Highlight the general performance
general_index = sorted_df.index[0]
general_pvalue = sorted_df.loc[general_index, 'pvalue']
general_train = sorted_df.loc[general_index, 'Train_best_model']
general_test = sorted_df.loc[general_index, 'Test_best_model']

plt.scatter(general_pvalue, general_train, color='darkgreen', s=150, label='General Performance (Train)', edgecolor='black', zorder=6)
plt.scatter(general_pvalue, general_test, color='darkorange', s=150, label='General Performance (Test)', edgecolor='black', zorder=6)

# Annotate the general performance with p-value, train, and test values
plt.text(general_pvalue, general_train, f'p={general_pvalue:.4g}\nTrain={general_train:.4g}', ha='left', va='bottom', fontsize=9, color='darkgreen')
plt.text(general_pvalue, general_test, f'p={general_pvalue:.4g}\nTest={general_test:.4g}', ha='left', va='top', fontsize=9, color='darkorange')

# Add labels and legend
plt.xlabel('p-value')
plt.ylabel('Model Performance')
plt.title('Train vs Test Best Models')
plt.legend()
plt.show()
 




print("2. Reporting Generalized Performance:\n")
df = divided_result.copy()
df['Difference'] = abs(df['Train_best_model'] - df['Test_best_model'])
df['Sum'] = df['Train_best_model'] + df['Test_best_model']
sorted_df = df.sort_values(by=['Sum', 'Difference'], ascending=[False, True])
print(sorted_df.iloc[0].to_markdown())


print("3. Reporting the correlation of hyperparameters and the performance of 'Train_null_model', 'Train_pure_prs', 'Train_best_model', 'Test_pure_prs', 'Test_null_model', and 'Test_best_model':\n")

print("3. For string hyperparameters, we used one-hot encoding to find the correlation between string hyperparameters and 'Train_null_model', 'Train_pure_prs', 'Train_best_model', 'Test_pure_prs', 'Test_null_model', and 'Test_best_model'.")

print("3. We performed this analysis for those hyperparameters that have more than one unique value.")

correlation_columns = [
 'Train_null_model', 'Train_pure_prs', 'Train_best_model',
 'Test_pure_prs', 'Test_null_model', 'Test_best_model'
]

hyperparams = [col for col in divided_result.columns if len(divided_result[col].unique()) > 1]
hyperparams = list(set(hyperparams+correlation_columns))
 
# Separate numeric and string columns
numeric_hyperparams = [col for col in hyperparams if pd.api.types.is_numeric_dtype(divided_result[col])]
string_hyperparams = [col for col in hyperparams if pd.api.types.is_string_dtype(divided_result[col])]


# Encode string columns using one-hot encoding
divided_result_encoded = pd.get_dummies(divided_result, columns=string_hyperparams)

# Combine numeric hyperparams with the new one-hot encoded columns
encoded_columns = [col for col in divided_result_encoded.columns if col.startswith(tuple(string_hyperparams))]
hyperparams = numeric_hyperparams + encoded_columns
 

# Calculate correlations
correlations = divided_result_encoded[hyperparams].corr()
 
# Display correlation of hyperparameters with train/test performance columns
hyperparam_correlations = correlations.loc[hyperparams, correlation_columns]
 
hyperparam_correlations = hyperparam_correlations.fillna(0)

# Plotting the correlation heatmap
plt.figure(figsize=(12, 8))
ax = sns.heatmap(hyperparam_correlations, annot=True, cmap='viridis', fmt='.2f', cbar=True)
ax.set_xticklabels(ax.get_xticklabels(), rotation=90, ha='right')

# Rotate y-axis labels to horizontal
#ax.set_yticklabels(ax.get_yticklabels(), rotation=0, va='center')

plt.title('Correlation of Hyperparameters with Train/Test Performance')
plt.show() 

sns.set_theme(style="whitegrid")  # Choose your preferred style
pairplot = sns.pairplot(divided_result_encoded[hyperparams],hue = 'Test_best_model', palette='viridis')

# Adjust the figure size
pairplot.fig.set_size_inches(15, 15)  # You can adjust the size as needed

for ax in pairplot.axes.flatten():
    ax.set_xlabel(ax.get_xlabel(), rotation=90, ha='right')  # X-axis labels vertical
    #ax.set_ylabel(ax.get_ylabel(), rotation=0, va='bottom')  # Y-axis labels horizontal

# Show the plot
plt.show()

1. Reporting Based on Best Training Performance:

|                   | 19                    |
|:------------------|:----------------------|
| clump_p1          | 1.0                   |
| clump_r2          | 0.1                   |
| clump_kb          | 200.0                 |
| p_window_size     | 200.0                 |
| p_slide_size      | 50.0                  |
| p_LD_threshold    | 0.25                  |
| pvalue            | 1.0                   |
| SDPR_M_val        | 1000.0                |
| SDPR_opt_llk_val  | 1.0                   |
| SDPR_iter_val_val | 1000.0                |
| SDPR_burn_val     | 200.0                 |
| SDPR_thin_val     | 5.0                   |
| SDPR_r2_val       | 0.1                   |
| SDPR_a_val        | 0.1                   |
| SDPR_c_val        | 1.0                   |
| SDPR_a0k_val      | 0.5                   |
| SDPR_b0k_val      | 0.5                   |
| numberofpca       | 6.0                   |
| tempalpha         | 0.1                   |
| l1weight          | 0.1                   |
| Train_pure_prs    | 4.49155982777949e-06  |
| Train_null_model  | 0.233899717941524     |
| Train_best_model  | 0.3641430010498434    |
| Test_pure_prs     | 5.093185732385486e-06 |
| Test_null_model   | 0.16758763481598576   |
| Test_best_model   | 0.31252760638907595   |
| SDPR_referenceset | custom                |

2. Reporting Generalized Performance:

|                   | 19                    |
|:------------------|:----------------------|
| clump_p1          | 1.0                   |
| clump_r2          | 0.1                   |
| clump_kb          | 200.0                 |
| p_window_size     | 200.0                 |
| p_slide_size      | 50.0                  |
| p_LD_threshold    | 0.25                  |
| pvalue            | 1.0                   |
| SDPR_M_val        | 1000.0                |
| SDPR_opt_llk_val  | 1.0                   |
| SDPR_iter_val_val | 1000.0                |
| SDPR_burn_val     | 200.0                 |
| SDPR_thin_val     | 5.0                   |
| SDPR_r2_val       | 0.1                   |
| SDPR_a_val        | 0.1                   |
| SDPR_c_val        | 1.0                   |
| SDPR_a0k_val      | 0.5                   |
| SDPR_b0k_val      | 0.5                   |
| numberofpca       | 6.0                   |
| tempalpha         | 0.1                   |
| l1weight          | 0.1                   |
| Train_pure_prs    | 4.49155982777949e-06  |
| Train_null_model  | 0.233899717941524     |
| Train_best_model  | 0.3641430010498434    |
| Test_pure_prs     | 5.093185732385486e-06 |
| Test_null_model   | 0.16758763481598576   |
| Test_best_model   | 0.31252760638907595   |
| SDPR_referenceset | custom                |
| Difference        | 0.05161539466076742   |
| Sum               | 0.6766706074389193    |
3. Reporting the correlation of hyperparameters and the performance of 'Train_null_model', 'Train_pure_prs', 'Train_best_model', 'Test_pure_prs', 'Test_null_model', and 'Test_best_model':

3. For string hyperparameters, we used one-hot encoding to find the correlation between string hyperparameters and 'Train_null_model', 'Train_pure_prs', 'Train_best_model', 'Test_pure_prs', 'Test_null_model', and 'Test_best_model'.
3. We performed this analysis for those hyperparameters that have more than one unique value.